https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17440 Lustre <p>We combined a OS update from TOSS 4.6-6 to TOSS 4.7-2.1 with a move from lustre 2.14 to 2.15 (2.14.0_22.llnl-1 to lustre-2.15.4_1.llnl-1).</p> <p>A few hours later we began to see this error, and eventually saw it on all 12 asp server nodes.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2024-01-16 23:13:19 [39638.886090] Lustre: aspls3-OST0004: Client 8f15405e-d4cc-cf3c-7534-051e7352cf50 (at 192.168.128.24@o2ib35) reconnecting 2024-01-16 23:13:19 [39638.896879] Lustre: Skipped 98 previous similar messages 2024-01-16 23:13:34 [39654.404520] LNetError: 165557:0:(peer.c:2194:lnet_destroy_peer_ni_locked()) ASSERTION( list_empty(&amp;lpni-&gt;lpni_peer_nis) ) failed: 2024-01-16 23:13:34 [39654.416271] LNetError: 165557:0:(peer.c:2194:lnet_destroy_peer_ni_locked()) LBUG 2024-01-16 23:13:34 [39654.423671] Pid: 165557, comm: kiblnd_sd_00_01 4.18.0-513.9.1.1toss.t4.x86_64 #1 SMP Wed Nov 29 11:04:55 PST 2023 2024-01-16 23:13:34 [39654.433921] Call Trace TBD: 2024-01-16 23:13:34 [39654.436731] [&lt;0&gt;] libcfs_call_trace+0x6f/0xa0 [libcfs] 2024-01-16 23:13:34 [39654.441888] [&lt;0&gt;] lbug_with_loc+0x3f/0x70 [libcfs] 2024-01-16 23:13:34 [39654.446688] [&lt;0&gt;] lnet_destroy_peer_ni_locked+0x44d/0x4e0 [lnet] 2024-01-16 23:13:34 [39654.452722] [&lt;0&gt;] lnet_handle_find_routed_path+0x86c/0xee0 [lnet] 2024-01-16 23:13:34 [39654.458845] [&lt;0&gt;] lnet_select_pathway+0xb95/0x16c0 [lnet] 2024-01-16 23:13:34 [39654.464265] [&lt;0&gt;] lnet_send+0x6d/0x1e0 [lnet] 2024-01-16 23:13:34 [39654.468646] [&lt;0&gt;] lnet_parse_local+0x3ed/0xdd0 [lnet] 2024-01-16 23:13:34 [39654.473721] [&lt;0&gt;] lnet_parse+0xd7d/0x1490 [lnet] 2024-01-16 23:13:34 [39654.478366] [&lt;0&gt;] kiblnd_handle_rx+0x30e/0x900 [ko2iblnd] 2024-01-16 23:13:34 [39654.483782] [&lt;0&gt;] kiblnd_scheduler+0x104b/0x10d0 [ko2iblnd] 2024-01-16 23:13:34 [39654.489363] [&lt;0&gt;] kthread+0x14c/0x170 2024-01-16 23:13:34 [39654.493030] [&lt;0&gt;] ret_from_fork+0x1f/0x40 2024-01-16 23:13:34 [39654.497050] Kernel panic - not syncing: LBUG 2024-01-16 23:13:34 [39654.501320] CPU: 47 PID: 165557 Comm: kiblnd_sd_00_01 Kdump: loaded Tainted: P           OE  X --------- -  - 4.18.0-513.9.1.1toss.t\ 4.x86_64 #1 2024-01-16 23:13:34 [39654.514172] Hardware name: Supermicro SSG-229P-DN2R24264-LL013/X11DSN-TS, BIOS 3.4 11/04/2020 2024-01-16 23:13:34 [39654.522683] Call Trace: 2024-01-16 23:13:34 [39654.525137]  dump_stack+0x41/0x60 2024-01-16 23:13:34 [39654.528457]  panic+0xe7/0x2ac 2024-01-16 23:13:34 [39654.531429]  ? ret_from_fork+0x1f/0x40 2024-01-16 23:13:34 [39654.535182]  lbug_with_loc.cold.8+0x18/0x18 [libcfs] 2024-01-16 23:13:34 [39654.540156]  lnet_destroy_peer_ni_locked+0x44d/0x4e0 [lnet] 2024-01-16 23:13:35 [39654.545747]  lnet_handle_find_routed_path+0x86c/0xee0 [lnet] 2024-01-16 23:13:35 [39654.551423]  ? lnet_peer_ni_find_locked+0x14/0x30 [lnet] 2024-01-16 23:13:35 [39654.556753]  lnet_select_pathway+0xb95/0x16c0 [lnet] 2024-01-16 23:13:35 [39654.561735]  ? kiblnd_check_sends_locked+0x1a5/0x4a0 [ko2iblnd] 2024-01-16 23:13:35 [39654.567656]  lnet_send+0x6d/0x1e0 [lnet] 2024-01-16 23:13:35 [39654.571600]  lnet_parse_local+0x3ed/0xdd0 [lnet] 2024-01-16 23:13:35 [39654.576238]  lnet_parse+0xd7d/0x1490 [lnet] 2024-01-16 23:13:35 [39654.580438]  ? try_to_wake_up+0x1c2/0x4f0 2024-01-16 23:13:35 [39654.584454]  kiblnd_handle_rx+0x30e/0x900 [ko2iblnd] 2024-01-16 23:13:35 [39654.589427]  ? __wake_up_common+0x7a/0x190 2024-01-16 23:13:35 [39654.593526]  kiblnd_scheduler+0x104b/0x10d0 [ko2iblnd] 2024-01-16 23:13:35 [39654.598665]  ? finish_wait+0x90/0x90 2024-01-16 23:13:35 [39654.602245]  ? kiblnd_cq_event+0x80/0x80 [ko2iblnd] 2024-01-16 23:13:35 [39654.607125]  kthread+0x14c/0x170 2024-01-16 23:13:35 [39654.610357]  ? set_kthread_struct+0x50/0x50 2024-01-16 23:13:35 [39654.614544]  ret_from_fork+0x1f/0x40 2024-01-16 23:13:36 [    0.000000] Linux version 4.18.0-513.9.1.1toss.t4.x86_64 (mockbuild@builder2-x86.buildfarm.internal) (gcc version 8.5.0 20210514 (Re\ d Hat 8.5.0-20) (GCC)) #1 SMP Wed Nov 29 11:04:55 PST 2023</pre> </div></div> TOSS 4.7-2.1 <br/> lustre 2.15.4_1.llnl-1 <br/> on lustre server asp LU-17440

after move from 2.14 to 2.15: LNetError: 31941:0:(peer.c:2194:lnet_destroy_peer_ni_locked()) ASSERTION( list_empty(&lpni->lpni_peer_nis) )

Bug Major Open Unresolved Serguei Smirnov Gian-Carlo Defazio llnl topllnl Wed, 17 Jan 2024 20:02:06 +0000 Wed, 7 Feb 2024 21:21:04 +0000 Lustre 2.15.4 Lustre 2.16.0 0 5 <p>For my notes the local issue is TOSS-6214</p> <p>The asp cluster has 12 server nodes, asp<span class="error">&#91;1-12&#93;</span>. asp<span class="error">&#91;1-4&#93;</span> are MDT<span class="error">&#91;0000-0003&#93;</span>, asp<span class="error">&#91;5-12&#93;</span> are OST<span class="error">&#91;0000-0007&#93;</span></p> <p>The nids are 172.19.1.<span class="error">&#91;141-152&#93;</span>@o2ib100 so the last octet of the IPV4 is (&lt;node_number&gt; + 140)</p> <p> </p> <p>I will also include some logs for orelic. orelic<span class="error">&#91;2-5&#93;</span> were rebooted from a 2.12 image to a 2.15 image from about 2024-01-16 (15:00 - 17:00). orelic is a ib to tcp lustre router cluster.</p> <p>The nids for orelic<span class="error">&#91;2-5&#93;</span> are 172.19.2.<span class="error">&#91;22-25&#93;</span>@o2ib100 and 172.16.66.<span class="error">&#91;22-25&#93;</span>@tcp</p> <p> </p> <p>The console logs are in asp-orelic-console.tar.gz</p> <p>I can't explain yet how exactly this happened. Did you update all nodes to lustre-2.15.4_1.llnl-1 or just the servers? Please share a core file if you have it.</p> <p>I'm a bit suspicious of <a href="https://jira.whamcloud.com/browse/LU-17062" title="Prevent use after free following *_decref_locked() usage" class="issue-link" data-issue-key="LU-17062"><del>LU-17062</del></a> "lnet: Update lnet_peer_*_decref_locked usage" change. If you have room to experiment, it may be worth reverting this change to see if it helps. I'll work on reproducing locally, too.</p> <p>Thanks,</p> <p>Serguei.</p> <p> </p> <p> </p> <p> </p> <p>There's still a variety of lustre versions, 2.12, 2.14, and 2.15</p> <p>The only production server cluster we've put 2.15 on is asp. The TOSS 4 (RHEL 8) server clusters are running 2.14 (2.14.0_24.llnl-1). asp was running 2.14.0_22.llnl-1 before the update. The TOSS 3 (RHEL 7) server clusters run 2.12.</p> <p>Most client clusters are running 2.12, some experimental and test clusters run 2.15.</p> <p>The relic clusters (zrelic and orelic) run 2.12, but orelic has been 2.15 for a while for testing. It got rebooted back to 2.12 on the morning of 2024-01-16 and then put back to 2.15 that afternoon.</p> <p> </p> <p>So to be clear, the only 2.15 updates were asp (a server cluster) and orelic (a router cluster).</p> <p>orelic had been 2.15 previously, asp had only ever been 2.14.</p> <p> </p> <p>One of the first things I checked was to make sure we didn't leave out <a href="https://jira.whamcloud.com/browse/LU-17062" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-17062</a> in our 2.15_4-llnl branch. We'll look into testing with that patch removed.</p> <p> </p> <p>Unfortunately we didn't get any core files, despite all the LBUGs.</p> <p>I did a review of which fixes from Lustre master LLNL branch may be missing, but haven't found anything that would stand out as definitely related. </p> <p>I'd suggest though that you consider porting <a href="https://jira.whamcloud.com/browse/LU-16709" title="LNet: locking multiple NIDs of the same MR peer as primary results in incorrect representation" class="issue-link" data-issue-key="LU-16709"><del>LU-16709</del></a> "lnet: fix locking multiple NIDs of the MR peer". It fixes <a href="https://jira.whamcloud.com/browse/LU-14668" title="LNet: do discovery in the background" class="issue-link" data-issue-key="LU-14668"><del>LU-14668</del></a> "lnet: Lock primary NID logic" which LLNL branch does have.</p> <p>Thanks,</p> <p>Serguei.</p> <p>I was able to reproduce the bug.</p> <p>The original bug seems to have been between asp and the compute cluster ruby. ruby has 8 routers and is connected to asp:</p> <p>&lt;ruby_compute_nodes&gt; &lt;—&gt; o2ib39 &lt;—&gt; &lt;ruby_routers&gt; &lt;—&gt; o2ib100 &lt;—&gt; asp</p> <p>However a route is missing on asp. The routes to ruby should be 172.19.2.<span class="error">&#91;39-46&#93;</span>, but the the routes are 172.19.2.<span class="error">&#91;39-45&#93;</span>, so 172.19.2.46 is missing. I noticed that this nid appears right before a lot of the LBUGs in the console logs.</p> <p> </p> <p>I recreated this setup using test clusters mutt (compute) and garter (server).</p> <p>The garter nids for garter<span class="error">&#91;1-8&#93;</span> are 172.19.1.<span class="error">&#91;133-140&#93;</span>@o2ib100</p> <p>&lt;mutt_compute_nodes&gt; &lt;—&gt; o2ib44 &lt;—&gt; &lt;mutt_routers&gt; &lt;—&gt; o2ib100 &lt;—&gt; garter</p> <p>The mutt routers are mutt<span class="error">&#91;1-4&#93;</span> with nids 172.19.1.<span class="error">&#91;105-108&#93;</span>@o2ib100 and 192.168.128.<span class="error">&#91;1-4&#93;</span>@o2ib44. I removed the route through mutt4 (172.19.1.108@o2ib100) on all garter nodes so now the routes are 172.19.1.<span class="error">&#91;105-107&#93;</span>@o2ib100.</p> <p>I then pinged from a mutt compute node, mutt7 (192.168.128.7@o2ib44)  to garter3 (172.19.1.135@o2ib100) using lnetctl ping. I had to ping twice, I assume to round-robin my way to using mutt4, for which garter has not route.</p> <p>This caused the error on garter3. I had panic on LBUG turned off.</p> <p>I'll upload the debug logs for mutt7, the mutt router mutt<span class="error">&#91;1-4&#93;</span> and garter3 as mutt-garter-reproducer.tar.gz</p> <p>I haven't looked through these debug logs yet.</p> <p>I'll see if the patch recommendations you made make a difference.</p> <p>Also, we are currently in the configuration with a route to ruby missing on asp, but asp is running lustre-2.14.0_24.llnl-1.t4.x86_64 and we don't get this bug.</p> <p>I was unable to recreate the bug using our 2.14 branch lustre-2.14.0_24.llnl-1.t4.x86_64 or our 2.15 branch lustre 2.15.4_1.llnl-1 with the only changes being <a href="https://jira.whamcloud.com/browse/LU-17062" title="Prevent use after free following *_decref_locked() usage" class="issue-link" data-issue-key="LU-17062"><del>LU-17062</del></a> reverted. I've uploaded the debug files for these tests.</p> <p>I noticed that I used garter4 for the server in these examples, whereas for the bug reproducer I used garter3.</p> <p>This is what I think is going wrong, do you agree?</p> <p>As noted before, without <a href="https://jira.whamcloud.com/browse/LU-17062" title="Prevent use after free following *_decref_locked() usage" class="issue-link" data-issue-key="LU-17062"><del>LU-17062</del></a>, the problem goes away.</p> <p>For the code lines below, I'm referring to where we added <a href="https://jira.whamcloud.com/browse/LU-17062" title="Prevent use after free following *_decref_locked() usage" class="issue-link" data-issue-key="LU-17062"><del>LU-17062</del></a> in our 2.15.4_1-llnl branch <a href="https://github.com/LLNL/lustre/commit/2e2719341590edef547e16b4051684781cf1b795" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/commit/2e2719341590edef547e16b4051684781cf1b795</a></p> <p>The debug data is in mutt-garter-debug-2fcd4d27f.tar.gz in the dk.TOSS-6214.garter3.lustre-2.15.4_1.llnl_3_g2fcd4d2-1.t4.x86_64.1706816872 file. Lines 737 to 771<br/> The debug patch version of lustre that was running on garter3 is at <a href="https://github.com/LLNL/lustre/tree/debug_print-lnet_handle_find_routed_path" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/tree/debug_print-lnet_handle_find_routed_path</a></p> <p>The setup is<br/> server garter3: 172.19.1.135@o2ib100<br/> routers mutt<span class="error">&#91;1-4&#93;</span>: 172.19.1.<span class="error">&#91;105-108&#93;</span>@o2ib100 and 192.168.128.<span class="error">&#91;1-4&#93;</span>@o2ib44<br/> client mutt7: 192.168.128.7@o2ib44</p> <p>garter3 &lt;—&gt; o2ib100 &lt;—&gt; mutt<span class="error">&#91;1-4&#93;</span> &lt;—&gt; o2ib44 &lt;—&gt; mutt7</p> <p>garter3 has routes to o2ib44 via mutt<span class="error">&#91;1-3&#93;</span>, mutt7 has routes to o2ib100 via mutt<span class="error">&#91;1-4&#93;</span>, so the is an asymmetry regarding mutt4<br/> mutt7 pings garter3 via mutt4</p> <p>Looking at function lnet_handle_find_routed_path():</p> <p>at line 2182, there is a call to lnet_peer_ni_find_locked()<br/> the call stack is lnet_handle_find_routed_path() -&gt; lnet_peer_ni_find_locked() -&gt; lnet_get_peer_ni_locked() -&gt; lnet_peer_ni_addref_locked()<br/> The return value is gwni. If gwni is not NULL, it is the gateway NI. In the debug logs for garter3, I see the nid for mutt4 when gwni is returned.</p> <p>The refcount for mutt4 is incremented due to lnet_peer_ni_addref_locked() being in the call stack above.</p> <p>before <a href="https://jira.whamcloud.com/browse/LU-17062" title="Prevent use after free following *_decref_locked() usage" class="issue-link" data-issue-key="LU-17062"><del>LU-17062</del></a>, if gwni is not NULL, the refcount for mutt4 is decremented almost immediately at line 2184<br/> after <a href="https://jira.whamcloud.com/browse/LU-17062" title="Prevent use after free following *_decref_locked() usage" class="issue-link" data-issue-key="LU-17062"><del>LU-17062</del></a>, the decrement happens at the end of the function.</p> <p>at line 2291, net_find_route_locked() is called. In the debug logs I can see the value of gwni-&gt;lpni_nid change after this call.<br/> It changes from the nid of mutt4 to the nid of another mutt router that garter3 does have a route through.</p> <p>before the <a href="https://jira.whamcloud.com/browse/LU-17062" title="Prevent use after free following *_decref_locked() usage" class="issue-link" data-issue-key="LU-17062"><del>LU-17062</del></a>, there were no other calls to lnet_peer_ni_decref_locked()<br/> after <a href="https://jira.whamcloud.com/browse/LU-17062" title="Prevent use after free following *_decref_locked() usage" class="issue-link" data-issue-key="LU-17062"><del>LU-17062</del></a> there is a call to lnet_peer_ni_decref_locked() if gwni was not NULL when it was modified at line 2182<br/> and hasn't subsequently become NULL. It shouldn't have become NULL if there's some NI that garter3 can use to get to o2ib44, which<br/> there is in this case.</p> <p>after <a href="https://jira.whamcloud.com/browse/LU-17062" title="Prevent use after free following *_decref_locked() usage" class="issue-link" data-issue-key="LU-17062"><del>LU-17062</del></a> at line 2365 lnet_peer_ni_decref_locked() is called on the a router with a refcount of 1 (assuming the gwni changed values due to net_find_route_locked()). The refcount goes to 0 (or would go to 0) so lnet_destroy_peer_ni_locked() is called. lnet_destroy_peer_ni_locked() has asserts to make sure something like this doesn't happen, so the asserts fail.</p> <p>Thanks <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=defazio" class="user-hover" rel="defazio">defazio</a> for tracking this down! I think your analysis is correct: indeed it looks like essentially we decref on what we didn't addref here. </p> <p>(I wonder if this can be reproduced with just two routers, for the sake of adding a testcase.)</p> <p>Are you planning to push a patch for this or would you like me to?</p> <p> </p> <p>I'll push a patch. I'm not entirely sure what the best way to go about this is, but I guess I'll find out in the code reviews.</p> <p>I'll run a test with 2 routers to see if I can reproduce it.</p> <p>Another weird thing I noticed was that it didn't reproduce on a different test cluster. That cluster, slag, isn't the same hardware as asp and garter. So I'm gonna go back and confirm that I didn't just have things set up wrong. However, the bug that I found doesn't look hardware specific.</p> <p>The bug does reproduce with 2 routers. I tested where mutt7 has routes through mutt<span class="error">&#91;1,2&#93;</span> and garter3 has a route through mutt1. The debug files are in mutt-garter-debug-2routers.tar.gz</p> <p> </p> <p>The bug does not reproduce in the case that garter3 has no routes to o2ib44. The debug files are in mutt-garter-debug-garter3_no_routes.tar.gz</p> <p>"Gian-Carlo DeFazio &lt;defazio1@llnl.gov&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53896" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53896</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17440" title="after move from 2.14 to 2.15: LNetError: 31941:0:(peer.c:2194:lnet_destroy_peer_ni_locked()) ASSERTION( list_empty(&amp;lpni-&gt;lpni_peer_nis) ) " class="issue-link" data-issue-key="LU-17440">LU-17440</a> lnet: prevent errorneous decref for asym route<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8466a612e1bf5ceda0e3bf32a5eada2277cbbc75</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ssmirnov" class="user-hover" rel="ssmirnov">ssmirnov</a><br/> I've gone with the patch that Chris Horn recommended. As for the test case, you mentioned using just two routers. Is there a vm configuration that already has 2 routers setup? Or do I need to set things up from within a test script like sanity-lnet.sh?</p> <p>I was hoping it would be possible to go with sanity-lnet.sh. There are some routing tests there already, but I don't think there's a test which uses 2 routers.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=hornc" class="user-hover" rel="hornc">hornc</a> can correct me if I'm wrong, but I think it should be possible to basically follow the existing pattern with new test if we make sure there are enough test nodes allocated.</p> Related Development Rank 1|i047zz: Rank (Obsolete) 9223372036854775807 Severity