https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5568 Lustre <p> 105.976884] Lustre: Lustre: Build Version: v2_6_51_0-g16c7568-CHANGED-3.10.1.el7_lustre<br/> [ 105.990490] LNetError: 2145:0:(socklnd.c:2660:ksocknal_enumerate_interfaces()) Can't find any usable interfaces<br/> [ 106.990120] LNetError: 105-4: Error -100 starting up LNI tcp<br/> [ 106.992703] LNetError: 2145:0:(api-ni.c:823:lnet_unprepare()) ASSERTION( list_empty(&amp;the_lnet.ln_nis) ) failed: <br/> [ 106.994560] LNetError: 2145:0:(api-ni.c:823:lnet_unprepare()) LBUG<br/> [ 106.994561] Pid: 2145, comm: modprobe<br/> [ 106.994561] \x0aCall Trace:<br/> [ 106.994574] <span class="error">&#91;&lt;ffffffffa044f853&gt;&#93;</span> libcfs_debug_dumpstack+0x53/0x80 <span class="error">&#91;libcfs&#93;</span><br/> [ 106.994578] <span class="error">&#91;&lt;ffffffffa044fdf5&gt;&#93;</span> lbug_with_loc+0x45/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> [ 106.994585] <span class="error">&#91;&lt;ffffffffa04f3267&gt;&#93;</span> lnet_unprepare+0x297/0x340 <span class="error">&#91;lnet&#93;</span><br/> [ 106.994587] <span class="error">&#91;&lt;ffffffffa04f3b5c&gt;&#93;</span> LNetNIInit+0x25c/0x3e0 <span class="error">&#91;lnet&#93;</span><br/> [ 106.994592] <span class="error">&#91;&lt;ffffffff81061bc6&gt;&#93;</span> ? put_online_cpus+0x56/0x80<br/> [ 106.994631] <span class="error">&#91;&lt;ffffffffa0983000&gt;&#93;</span> ? init_module+0x0/0x1000 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 106.994658] <span class="error">&#91;&lt;ffffffffa081310c&gt;&#93;</span> ptlrpc_ni_init+0x2c/0x1a0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 106.994679] <span class="error">&#91;&lt;ffffffffa0983000&gt;&#93;</span> ? init_module+0x0/0x1000 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 106.994703] <span class="error">&#91;&lt;ffffffffa0813291&gt;&#93;</span> ptlrpc_init_portals+0x11/0xf0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 106.994722] <span class="error">&#91;&lt;ffffffffa0983000&gt;&#93;</span> ? init_module+0x0/0x1000 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 106.994739] <span class="error">&#91;&lt;ffffffffa09831c4&gt;&#93;</span> init_module+0x1c4/0x1000 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 106.994742] <span class="error">&#91;&lt;ffffffff810020e2&gt;&#93;</span> do_one_initcall+0xe2/0x190<br/> [ 106.994744] <span class="error">&#91;&lt;ffffffff810ca7fb&gt;&#93;</span> load_module+0x129b/0x1a90<br/> [ 106.994745] <span class="error">&#91;&lt;ffffffff812da590&gt;&#93;</span> ? ddebug_dyndbg_module_param_cb+0x0/0x60<br/> [ 106.994747] <span class="error">&#91;&lt;ffffffff810c7133&gt;&#93;</span> ? copy_module_from_fd.isra.43+0x53/0x150<br/> [ 106.994748] <span class="error">&#91;&lt;ffffffff810cb1a6&gt;&#93;</span> SyS_finit_module+0xa6/0xd0<br/> [ 106.994750] <span class="error">&#91;&lt;ffffffff815f2119&gt;&#93;</span> system_call_fastpath+0x16/0x1b<br/> [ 106.994750] <br/> [ 106.995032] Kernel panic - not syncing: LBUG<br/> [ 106.995034] CPU: 3 PID: 2145 Comm: modprobe Tainted: GF O-------------- 3.10.1.el7_lustre #1<br/> [ 106.995034] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013<br/> [ 106.995036] ffffffffa0474d0d 00000000d711e588 ffff880036601bf0 ffffffff815e19ba<br/> [ 106.995037] ffff880036601c70 ffffffff815db549 ffffffff00000008 ffff880036601c80<br/> [ 106.995037] ffff880036601c20 00000000d711e588 ffffffffa051574f 0000000000000000<br/> [ 106.995038] Call Trace:<br/> [ 106.995052] <span class="error">&#91;&lt;ffffffff815e19ba&gt;&#93;</span> dump_stack+0x19/0x1b<br/> [ 106.995055] <span class="error">&#91;&lt;ffffffff815db549&gt;&#93;</span> panic+0xd8/0x1e7<br/> [ 106.995062] <span class="error">&#91;&lt;ffffffffa044fe5b&gt;&#93;</span> lbug_with_loc+0xab/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> [ 106.995067] <span class="error">&#91;&lt;ffffffffa04f3267&gt;&#93;</span> lnet_unprepare+0x297/0x340 <span class="error">&#91;lnet&#93;</span><br/> [ 106.995070] <span class="error">&#91;&lt;ffffffffa04f3b5c&gt;&#93;</span> LNetNIInit+0x25c/0x3e0 <span class="error">&#91;lnet&#93;</span><br/> [ 106.995072] <span class="error">&#91;&lt;ffffffff81061bc6&gt;&#93;</span> ? put_online_cpus+0x56/0x80<br/> [ 106.995088] <span class="error">&#91;&lt;ffffffffa0983000&gt;&#93;</span> ? 0xffffffffa0982fff<br/> [ 106.995113] <span class="error">&#91;&lt;ffffffffa081310c&gt;&#93;</span> ptlrpc_ni_init+0x2c/0x1a0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 106.995117] <span class="error">&#91;&lt;ffffffffa0983000&gt;&#93;</span> ? 0xffffffffa0982fff<br/> [ 106.995138] <span class="error">&#91;&lt;ffffffffa0813291&gt;&#93;</span> ptlrpc_init_portals+0x11/0xf0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 106.995141] <span class="error">&#91;&lt;ffffffffa0983000&gt;&#93;</span> ? 0xffffffffa0982fff<br/> [ 106.995179] <span class="error">&#91;&lt;ffffffffa09831c4&gt;&#93;</span> init_module+0x1c4/0x1000 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 106.995181] <span class="error">&#91;&lt;ffffffff810020e2&gt;&#93;</span> do_one_initcall+0xe2/0x190<br/> [ 106.995182] <span class="error">&#91;&lt;ffffffff810ca7fb&gt;&#93;</span> load_module+0x129b/0x1a90<br/> [ 106.995185] <span class="error">&#91;&lt;ffffffff812da590&gt;&#93;</span> ? ddebug_proc_write+0xf0/0xf0<br/> [ 106.995186] <span class="error">&#91;&lt;ffffffff810c7133&gt;&#93;</span> ? copy_module_from_fd.isra.43+0x53/0x150<br/> [ 106.995187] <span class="error">&#91;&lt;ffffffff810cb1a6&gt;&#93;</span> SyS_finit_module+0xa6/0xd0<br/> [ 106.995189] <span class="error">&#91;&lt;ffffffff815f2119&gt;&#93;</span> system_call_fastpath+0x16/0x1b</p> LU-5568

kernel crash when when network initialization failed

Bug Critical Resolved Fixed Amir Shehata Wang Shilong MB patch Mon, 1 Sep 2014 15:08:05 +0000 Wed, 17 Dec 2014 19:16:58 +0000 Wed, 17 Dec 2014 19:16:58 +0000 Lustre 2.7.0 Lustre 2.7.0 0 8 <p><a href="http://review.whamcloud.com/#/c/11718/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11718/</a></p> <p>Amir</p> <p>Could you please review this patch?</p> <p>Thanks</p> <p>Peter</p> <p>The bug seemed to be introduced by commit 92c51841c50cc4061c20b277d3f7c4468f2a80cc. While the proposed patch fixed the symptom, it left the underlying API inconsistency unfixed: the lnet internal initialization APIs are somewhat transaction-like in that if a init function fails it cleanups itself so the caller don't have to call the corresponding fini function, e.g. if lnet_prepare() fails there's no need to call lnet_unprepare().</p> <p>But with 92c51841c50cc4061c20b277d3f7c4468f2a80cc, lnet_shutdown_lndnis() was removed from lnet_startup_lndnis(), so now the callers of lnet_startup_lndnis() will be responsible to clean up if lnet_startup_lndnis() has failed, which breaks the convention that init() functions clean up by themselves on failures. This kind of inconsistency will cause us troubles in the future. I'd suggest to:<br/> 1. Move lnet_shutdown_lndnis() back to lnet_startup_lndnis() so that lnet_startup_lndnis() will cleanup itself.<br/> 2. Move the code in lnet_startup_lndnis() that starts a single NI into a new function e.g. startup_a_single_ni().<br/> 3. Make lnet_dyn_add_ni() call startup_a_single_ni() instead of lnet_startup_lndnis().</p> <p>Wang, do you have plans of addressing Isaac's points in your patch?</p> <p>Hi Oleg Drokin,</p> <p>Yeah, i did address lsaac's comment!</p> <p>Hi Isaac Huang,</p> <p>Thanks very much for your comments, could you take a look and give me some response<br/> about last your comments:</p> <p>"When we goto failed here, the ni is no longer on the nilist, then where is the ni going to be freed?"</p> <p>My reply: lnet_shutdown_lndnis() could do that?</p> <p>btw, i am not sure why i reply comments at redmine, it did not give email or something...</p> <p>lnet_shutdown_lndnis() will not free the ni if the failure happened early, i.e. the ni hasn't been added to any global lists. For example, if it failed in the first few conditional checks in lnet_startup_lndni(), lnet_shutdown_lndnis() will not be called at all.</p> <p>I think the fix should be considered together with <a href="https://jira.whamcloud.com/browse/LU-5734" title="LNet dynamic control: lnet_dyn_add_ni() can&#39;t clean up failed NI in some cases" class="issue-link" data-issue-key="LU-5734"><del>LU-5734</del></a> - the two are closely related.</p> <p>Hi lsaac Huang,</p> <p>I am sorry for bothering you, could you please help review new version<br/> and give me some feedbacks, thank you very much!</p> <p>Best regards,<br/> Wang Shilong</p> <p>Landed for 2.7</p> <p>Patch reverted. Amir could you please look into this issue? Thanks</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12512/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12512/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5568" title="kernel crash when when network initialization failed" class="issue-link" data-issue-key="LU-5568"><del>LU-5568</del></a> lnet: fix kernel crash when network failed to start<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 66e9055b23433bd0aa8da5e49f3b665fb1b95532</p> Duplicate LU-5664 LU-5884 Related LU-2456 LU-5734 Development Epic/Theme lnet Rank 1|hzwv1j: Rank (Obsolete) 15529 Severity