https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7569 Lustre <p>During testing we lost one of the IB leaf switches which caused all of our lustre router to crash with the following error:</p> <p>2015-12-11T10:53:29.539273-05:00 c0-0c0s2n3 LNetError: 4675:0:(o2iblnd.c:399:kiblnd_find_peer_locked()) ASSERTION( peer-&gt;ibp_connecting &gt; 0 || peer-&gt;ibp_accepting &gt; 0 || !list_empty(&amp;peer-&gt;ibp_conns) ) failed:<br/> 2015-12-11T10:53:29.539305-05:00 c0-0c0s2n3 LNetError: 4675:0:(o2iblnd.c:399:kiblnd_find_peer_locked()) LBUG<br/> 2015-12-11T10:53:29.539313-05:00 c0-0c0s2n3 Pid: 4675, comm: kgnilnd_sd_02<br/> 2015-12-11T10:53:29.539319-05:00 c0-0c0s2n3 Call Trace:<br/> 2015-12-11T10:53:29.539324-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff81006651&gt;&#93;</span> try_stack_unwind+0x161/0x1a0<br/> 2015-12-11T10:53:29.539332-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff81004eb9&gt;&#93;</span> dump_trace+0x89/0x430<br/> 2015-12-11T10:53:29.539339-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa025bac0&gt;&#93;</span> lbug_with_loc+0x90/0x1d0 <span class="error">&#91;libcfs&#93;</span><br/> 2015-12-11T10:53:29.539348-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa036f32b&gt;&#93;</span> kiblnd_find_peer_locked+0x14b/0x150 <span class="error">&#91;ko2iblnd&#93;</span><br/> 2015-12-11T10:53:29.539358-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa036f379&gt;&#93;</span> kiblnd_query+0x49/0x1c0 <span class="error">&#91;ko2iblnd&#93;</span><br/> 2015-12-11T10:53:29.539364-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02d5aee&gt;&#93;</span> lnet_post_send_locked+0x2ee/0x740 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539369-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02d84f0&gt;&#93;</span> lnet_send+0x6a0/0xcf0 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539375-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02cbe94&gt;&#93;</span> lnet_finalize+0x424/0x800 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539380-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03d256b&gt;&#93;</span> kgnilnd_recv+0x73b/0xdf0 <span class="error">&#91;kgnilnd&#93;</span><br/> 2015-12-11T10:53:29.539385-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02d432f&gt;&#93;</span> lnet_ni_recv+0xcf/0x330 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539389-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02dac26&gt;&#93;</span> lnet_parse+0x3c6/0xe40 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539394-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03d8111&gt;&#93;</span> kgnilnd_check_fma_rx+0x1af1/0x1f50 <span class="error">&#91;kgnilnd&#93;</span><br/> 2015-12-11T10:53:29.539406-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03dbbc4&gt;&#93;</span> kgnilnd_process_conns+0x554/0x15d0 <span class="error">&#91;kgnilnd&#93;</span><br/> 2015-12-11T10:53:29.539411-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03dcf1e&gt;&#93;</span> kgnilnd_scheduler+0x2de/0x5f0 <span class="error">&#91;kgnilnd&#93;</span><br/> 2015-12-11T10:53:29.539416-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff81067ace&gt;&#93;</span> kthread+0x9e/0xb0<br/> 2015-12-11T10:53:29.539421-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff81490074&gt;&#93;</span> kernel_thread_helper+0x4/0x10<br/> 2015-12-11T10:53:29.539427-05:00 c0-0c0s2n3 Kernel panic - not syncing: LBUG<br/> 2015-12-11T10:53:29.539432-05:00 c0-0c0s2n3 Pid: 4675, comm: kgnilnd_sd_02 Tainted: P 3.0.101-0.46.1_1.0502.8871-cray_gem_s #1<br/> 2015-12-11T10:53:29.539437-05:00 c0-0c0s2n3 Call Trace:<br/> 2015-12-11T10:53:29.539442-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff81006651&gt;&#93;</span> try_stack_unwind+0x161/0x1a0<br/> 2015-12-11T10:53:29.539447-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff81004eb9&gt;&#93;</span> dump_trace+0x89/0x430<br/> 2015-12-11T10:53:29.539452-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff810060bc&gt;&#93;</span> show_trace_log_lvl+0x5c/0x80<br/> 2015-12-11T10:53:29.539457-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff810060f5&gt;&#93;</span> show_trace+0x15/0x20<br/> 2015-12-11T10:53:29.539462-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff8148b31c&gt;&#93;</span> dump_stack+0x79/0x84<br/> 2015-12-11T10:53:29.539467-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff8148b3bb&gt;&#93;</span> panic+0x94/0x1da<br/> 2015-12-11T10:53:29.539473-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa025bbf1&gt;&#93;</span> lbug_with_loc+0x1c1/0x1d0 <span class="error">&#91;libcfs&#93;</span><br/> 2015-12-11T10:53:29.539479-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa036f32b&gt;&#93;</span> kiblnd_find_peer_locked+0x14b/0x150 <span class="error">&#91;ko2iblnd&#93;</span><br/> 2015-12-11T10:53:29.539484-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa036f379&gt;&#93;</span> kiblnd_query+0x49/0x1c0 <span class="error">&#91;ko2iblnd&#93;</span><br/> 2015-12-11T10:53:29.539489-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02d5aee&gt;&#93;</span> lnet_post_send_locked+0x2ee/0x740 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539494-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02d84f0&gt;&#93;</span> lnet_send+0x6a0/0xcf0 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539500-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02cbe94&gt;&#93;</span> lnet_finalize+0x424/0x800 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539505-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03d256b&gt;&#93;</span> kgnilnd_recv+0x73b/0xdf0 <span class="error">&#91;kgnilnd&#93;</span><br/> 2015-12-11T10:53:29.539511-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02d432f&gt;&#93;</span> lnet_ni_recv+0xcf/0x330 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539519-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa02dac26&gt;&#93;</span> lnet_parse+0x3c6/0xe40 <span class="error">&#91;lnet&#93;</span><br/> 2015-12-11T10:53:29.539526-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03d8111&gt;&#93;</span> kgnilnd_check_fma_rx+0x1af1/0x1f50 <span class="error">&#91;kgnilnd&#93;</span><br/> 2015-12-11T10:53:29.539532-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03dbbc4&gt;&#93;</span> kgnilnd_process_conns+0x554/0x15d0 <span class="error">&#91;kgnilnd&#93;</span><br/> 2015-12-11T10:53:29.539537-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03dcf1e&gt;&#93;</span> kgnilnd_scheduler+0x2de/0x5f0 <span class="error">&#91;kgnilnd&#93;</span><br/> 2015-12-11T10:53:29.539544-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff81067ace&gt;&#93;</span> kthread+0x9e/0xb0<br/> 2015-12-11T10:53:29.539550-05:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff81490074&gt;&#93;</span> kernel_thread_helper+0x4/0x10</p> LU-7569

IB leaf switch caused LNet routers to crash

Bug Major Resolved Fixed Doug Oucharek James A Simmons Wed, 16 Dec 2015 23:43:44 +0000 Tue, 24 Oct 2017 13:02:34 +0000 Mon, 18 Jan 2016 18:59:45 +0000 Lustre 2.8.0 Lustre 2.8.0 0 21 <p>Hi Amir,</p> <p>Could you please advise? Thank you.</p> <p>I've seen a handful of at least suspects ways these conditional checks for ibp_connecting, ibp_accepting and ibp_conns can be incorrect if things are slow to respond and connections get rejected. With the inclusion of <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> rejections can be common and I think this is exposing some of these problems with the reconnect logic and more so conn race patch <a href="http://review.whamcloud.com/14600/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14600/</a>.</p> <p>Liang Zhen (liang.zhen@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17661" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17661</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7569" title="IB leaf switch caused LNet routers to crash" class="issue-link" data-issue-key="LU-7569"><del>LU-7569</del></a> o2iblnd: multiple fixes for reconnection<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e4777765fc6e4bdc9a9331e139a5884261245eb0</p> <p>I've submitted a patch which could be helpful, <a href="http://review.whamcloud.com/17661" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17661</a> , but I have no environment to test it, so it is only for review for the time being.</p> <p>Liang I just rebooted a system with 17661 and we rebooted the leaf switch. It completely worked. No more routers OOPs on us. Thank you.</p> <p>James, did all the clients re-connect ok? None of them got stuck on reconnecting?</p> <p>Yes Doug they did all reconnect okay. I did find a problem with this patch tho. I found if I place the following in my modprobe configuration file I can crash my client nodes.</p> <p>options ko2iblnd timeout=100 credits=2560 ntx=5120 peer_credits=63 concurrent_sends=63 fmr_pool_size=1280 fmr_flush_trigger=1024 map_on_demand=64</p> <p>and then modprobe lnet;lctl net up</p> <p>You will then see the following back trace:<br/> Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.292250] LNetError: 20000:0:(o2iblnd_cb.c:1309:kiblnd_reconnect_pee r()) ASSERTION( peer-&gt;ibp_connecting == 1 ) failed: <br/> Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.303363] LNetError: 20000:0:(o2iblnd_cb.c:1309:kiblnd_reconnect_pee r()) LBUG Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.310739] Pid: 20000, comm: kiblnd_connd <br/> Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.314891] <br/> Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.314892] Call Trace: Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.318939] <span class="error">&#91;&lt;ffffffffa0740875&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.325962] <span class="error">&#91;&lt;ffffffffa0740e77&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span> <br/> Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.332212] <span class="error">&#91;&lt;ffffffffa08431c8&gt;&#93;</span> kiblnd_reconnect_peer+0x118/0x150 <span class="error">&#91;ko2iblnd&#93;</span> <br/> Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.339608] <span class="error">&#91;&lt;ffffffffa083aee0&gt;&#93;</span> kiblnd_destroy_conn+0x4c0/0x810 <span class="error">&#91;ko2iblnd&#93;</span> <br/> Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.346784] <span class="error">&#91;&lt;ffffffffa08486b1&gt;&#93;</span> kiblnd_connd+0xc1/0xbc0 <span class="error">&#91;ko2iblnd&#93;</span><br/> Dec 30 15:02:05 spoon17.ccs.ornl.gov kernel: [ 337.353248] <span class="error">&#91;&lt;ffffffff81064d00&gt;&#93;</span> ? default_wake_function+0x0/0x20 </p> <p>Liang, Doug have you been able to duplicate my crash?</p> <p>Just a status update on this patch: There are multiple problems with the reconnection code on o2iblnd which we are trying to address here (see list of related patches). As you can see from Liang's patch, significant changes are being made to the reconnection strategy to address them.</p> <p>At the moment, I know of two issues with the current version of this patch:</p> <p>1- Reconnections due to different negotiated parameters can cause an LBUG (what you are finding James)<br/> 2- When an LNet router reboots, an infinite loop of CONN RACE reconnects can ensue if the LNet router has the larger NID value.</p> <p>I'm working on number 2 with a customer who has run into this. My current theory is that the client tries to reconnect to the router while the router has not completely come up yet. If that connection attempt gets stuck (i.e. client never hears back from it), it can trigger never-ending reconnects.</p> <p>I'd like the patch for this ticket address the above 2 issues so we can kill off many problems at once here.</p> <p>I think I know why number 1 happens. The function kiblnd_check_reconnect() return right away if peer-&gt;ibp_connecting != 1. So for the checks to actually happen we need the condition peer-&gt;ibp_connecting == 1. But for some of the checks we end up incrementing ibp_connecting again. I think the logic is reversed from what it should be. Since we know peer-&gt;ibp_connecting == 1 on critical failure it should decremented. I'm testing this change now.</p> <p>My theory was wrong. Still crashes.</p> <p>Thanks to Jeremy he pointed out I was using a old patch so with my fix the latest version of the patch resolves problem 1 Doug listed. I haven't run into case 2 so no fix for that.</p> <p>That is great news! For issue 2, I have a theoretical fix which will be tested today. From the logs, I am seeing this pattern which causes issue 2:</p> <p>1- LNet router reboots (for any reason).<br/> 2- A client fails to transmit to the router and fails the connection. This cleans up the connection and peer structure.<br/> 3- Attempts to create active connections from the client to the router are continuously made until the router is back up.<br/> 4- I then see the router trying to create an active connection to the client.<br/> 5- For some unknown reason, the client seems to have a connection to the router stuck in a connecting state. <br/> 6- In this scenario. the client has the larger NID value so the client rejects router's connection attempt as CONN RACE.<br/> 7- The router does a reconnect, goto 5.</p> <p>Thus, we have an infinite loop caused by what appears to be a stuck connecting connection. The code assumes we will always hear back from connection attempts so there is no cleanup of the connection by connd. Why the connection is stuck is unknown to me, but the code should be robust enough to detect this and avoid this infinite loop (i.e. self-healing code).</p> <p>Note: I am only seeing this with mlx5-based cards.</p> <p>as the original patch has defects and been reverted from master, so the above patch is invalid now, I reimplemented the patch, which includes the original feature and improvements : <a href="http://review.whamcloud.com/#/c/17892/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/17892/</a></p> <p>For the stuck issue described by Doug, I'm not sure if it is a general issue, or just a bug in a particular version of mlx5 driver. So this patch didn't include the self-healing code mentioned by Doug. If you don't have the stuck issue, then probably you don't need the self-healing code at all.</p> <p>Doug since you don't have a solution just yet for the connection issue could you create a new patch on top of the new one posted by Liang.</p> <p>Ok, will do. I will create a new Jira ticket for that solution so this ticket is free to land Liang's latest patch and close.</p> <p>Perhaps this problem will not exist with Liang latest patch? Its worth a try.</p> <p>Jay: once this has landed to master (inspected, tested, etc) then it can be ported.</p> <p>I'm of the opinion that this should be a blocker. Currently without this work it is possible if a IB leaf switch reboots to take down all the LNet routers. Because of this it needs to be slated for 2.8 landing.</p> <p>Is this still true with 14600 reverted?</p> <p>I just tested this patch with 14600 reverted on our Cray system and the routers crashed again. So this is still a serious problem. If we lose a IB leaf switch we lose the entire file system. IMHO this should be a blocker.</p> <p>what's the crash looks like after reverting of 14600, is it OOM or another assertion?</p> <p>Hmmm. Doesn't appear to LNet related.</p> <p>2016-01-13T15:56:22.464787-05:00 c0-0c0s7n1 LustreError: 167-0: sultan-OST000e-osc-ffff880405221400: This client was evicted by sultan-OST000e; in progress operations using this service will fail.<br/> 2016-01-13T15:56:22.464804-05:00 c0-0c0s7n1 LustreError: Skipped 11 previous similar messages<br/> 2016-01-13T15:56:22.514072-05:00 c0-0c0s7n1 Lustre: 2541:0:(llite_lib.c:2628:ll_dirty_page_discard_warn()) sultan: dirty page discard: 10.37.248.67@o2ib1:/sultan/fid: <span class="error">&#91;0x20000a898:0x21:0x0&#93;</span>//stf008/scratch/jsimmons/test_ior/testfile.out may get corrupted (rc -108)<br/> 2016-01-13T15:56:22.756634-05:00 c0-0c0s7n1 Lustre: sultan-OST0025-osc-ffff880405221400: Connection restored to 10.37.248.70@o2ib1 (at 10.37.248.70@o2ib1)</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17892/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17892/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7569" title="IB leaf switch caused LNet routers to crash" class="issue-link" data-issue-key="LU-7569"><del>LU-7569</del></a> o2iblnd: avoid intensive reconnecting<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 9ab698e4d99103b2fecf19b0fd3f90d28723e9d1</p> <p>Landed for 2.8</p> <p>Jay, This patch is under review now. So, soon it will be landed to b2_7_fe also.</p> <p>Jay, it has been ported to 2.7 FE as: <a href="http://review.whamcloud.com/#/c/18051/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/18051/</a>. </p> <p>Doug, although the patch at <a href="http://review.whamcloud.com/#/c/18051/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/18051/</a>. <br/> says "b2_7_fe" under "branch", there is a red remark saying "Cannot Merge" under "Strategy."</p> <p>Probably the patch was not generated against b2_7_fe? I encountered non-trivia conflicts at<br/> both modified: lnet/klnds/o2iblnd/o2iblnd.h<br/> both modified: lnet/klnds/o2iblnd/o2iblnd_cb.c</p> Related LU-7390 LU-7646 LU-5718 LU-7210 LU-3322 LU-7314 LU-7676 Development Rank 1|hzxw3r: Rank (Obsolete) 9223372036854775807 Severity