https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7362 Lustre <p>While testing the latest Lustre pre-2.8 release on one of our Cray systems DVS was enabled by mistake on a router which LNet then crashed with the following backtrace:</p> <p>2015-10-27T11:39:24.142188-04:00 c0-1c1s1n1 Pid: 16118, comm: router_checker<br/> 2015-10-27T11:39:24.142197-04:00 c0-1c1s1n1 Call Trace:<br/> 2015-10-27T11:39:24.142213-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff81006651&gt;&#93;</span> try_stack_unwind+0x161/0x1a0<br/> 2015-10-27T11:39:24.142225-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff81004eb9&gt;&#93;</span> dump_trace+0x89/0x430<br/> 2015-10-27T11:39:24.142240-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffffa025b897&gt;&#93;</span> libcfs_debug_dumpstack+0x57/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 2015-10-27T11:39:24.142255-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffffa025bde7&gt;&#93;</span> lbug_with_loc+0x47/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> 2015-10-27T11:39:24.181560-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffffa02f29b6&gt;&#93;</span> lnet_router_checker+0x566/0x5a0 <span class="error">&#91;lnet&#93;</span><br/> 2015-10-27T11:39:24.181581-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff81067ace&gt;&#93;</span> kthread+0x9e/0xb0<br/> 2015-10-27T11:39:24.181609-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff81490074&gt;&#93;</span> kernel_thread_helper+0x4/0x10<br/> 2015-10-27T11:39:24.181616-04:00 c0-1c1s1n1 Kernel panic - not syncing: LBUG<br/> 2015-10-27T11:39:24.181627-04:00 c0-1c1s1n1 Pid: 16118, comm: router_checker Tainted: P 3.0.101-0.46.1_1.0502.8871-cray_gem_s #1<br/> 2015-10-27T11:39:24.211395-04:00 c0-1c1s1n1 Call Trace:<br/> 2015-10-27T11:39:24.211415-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff81006651&gt;&#93;</span> try_stack_unwind+0x161/0x1a0<br/> 2015-10-27T11:39:24.211422-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff81004eb9&gt;&#93;</span> dump_trace+0x89/0x430<br/> 2015-10-27T11:39:24.211476-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff810060bc&gt;&#93;</span> show_trace_log_lvl+0x5c/0x80<br/> 2015-10-27T11:39:24.211488-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff810060f5&gt;&#93;</span> show_trace+0x15/0x20<br/> 2015-10-27T11:39:24.211515-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff8148b31c&gt;&#93;</span> dump_stack+0x79/0x84<br/> 2015-10-27T11:39:24.211531-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff8148b3bb&gt;&#93;</span> panic+0x94/0x1da<br/> 2015-10-27T11:39:24.211560-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffffa025be4b&gt;&#93;</span> lbug_with_loc+0xab/0xc0 <span class="error">&#91;libcfs&#93;</span><br/> 2015-10-27T11:39:24.211579-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffffa02f29b6&gt;&#93;</span> lnet_router_checker+0x566/0x5a0 <span class="error">&#91;lnet&#93;</span><br/> 2015-10-27T11:39:24.211586-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff81067ace&gt;&#93;</span> kthread+0x9e/0xb0<br/> 2015-10-27T11:39:24.241857-04:00 c0-1c1s1n1 <span class="error">&#91;&lt;ffffffff81490074&gt;&#93;</span> kernel_thread_helper+0x4/0x10</p> <p>While DVS is a external utility on top of LNet it shouldn't be able to crash a LNet router.</p> Cray Routers running latest Lustre pre-2.8 LU-7362

During our larger scale testing DVS was accidentally started on a router which could LNet to kernel crash

Bug Minor Resolved Fixed Doug Oucharek James A Simmons Fri, 30 Oct 2015 14:39:20 +0000 Thu, 13 Oct 2016 17:07:03 +0000 Wed, 18 Nov 2015 01:39:44 +0000 Lustre 2.8.0 Lustre 2.8.0 0 8 <p>James: This appears to be from an LASSERT. Can you tell us which of the two possible asserts in the router checker routine this is coming from: </p> <p>LASSERT (the_lnet.ln_rc_state == LNET_RC_STATE_RUNNING);<br/> or<br/> LASSERT(the_lnet.ln_rc_state == LNET_RC_STATE_STOPPING);</p> <p>It seems likely to be the first case (check for RUNNING). Initial thought: we are starting up and launch the router checker thread. Before it can run, DVS stops LNet (for some reason). That changes state to STOPPING before the router checker thread starts and does that first assert.</p> <p>Personally, I hate asserts for checks like this. I'd like to address this ticket by removing that first assert and let the router checker loop terminate immediately because the state has changed (i.e. neither one of these asserts is really protecting us from anything and are not valid reasons for crashing a system).</p> <p>As DVS is a Cray tool, could someone at Cray comment on whether DVS would be stopping LNet immediately upon startup like this?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; sym the_lnet ffffffffa030cb60 (B) the_lnet [lnet] crash&gt; lnet_t ffffffffa030cb60 | grep ln_rc_state ln_rc_state = 2, </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lnet/include/lnet/lib-types.h:#define LNET_RC_STATE_SHUTDOWN 0 /* not started */ lnet/include/lnet/lib-types.h:#define LNET_RC_STATE_RUNNING 1 /* started up OK */ lnet/include/lnet/lib-types.h:#define LNET_RC_STATE_STOPPING 2 /* telling thread to stop */ </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2015-10-27T11:39:22.941791-04:00 c0-1c1s1n1 LNet: Added LNI 701@gni110 [16/8192/0/0] 2015-10-27T11:39:22.941799-04:00 c0-1c1s1n1 LNet: Added LNI 701@gni111 [16/8192/0/0] 2015-10-27T11:39:22.941805-04:00 c0-1c1s1n1 LNet: Added LNI 701@gni112 [16/8192/0/0] 2015-10-27T11:39:22.941812-04:00 c0-1c1s1n1 LNet: Added LNI 10.36.230.100@o2ib106 [63/2560/0/180] 2015-10-27T11:39:22.941818-04:00 c0-1c1s1n1 LNet: Added LNI 10.36.230.100@o2ib225 [63/2560/0/180] 2015-10-27T11:39:24.142128-04:00 c0-1c1s1n1 DVS: dvs_lnet_init: No network ID found on configured lnd (gni100) 2015-10-27T11:39:24.142174-04:00 c0-1c1s1n1 LNetError: 16118:0:(router.c:1241:lnet_router_checker()) ASSERTION( the_lnet.ln_rc_state == 1 ) failed: 2015-10-27T11:39:24.142181-04:00 c0-1c1s1n1 LNetError: 16118:0:(router.c:1241:lnet_router_checker()) LBUG </pre> </div></div> <p>DVS will call LNetNIFini() (stopping LNet) when it can't find the "network ID...on configured lnd"</p> <p>Thanks Matt. That seems to confirm my suspicion. I'm assuming that dvs_lnet_init() stops LNet because it cannot find the network ID it is expecting.</p> <p>So, I'm proposing to remove the offending assert. Rather than protecting us from anything (which it does not), it has become a problem.</p> <p>Doug Oucharek (doug.s.oucharek@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/17003" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17003</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7362" title="During our larger scale testing DVS was accidentally started on a router which could LNet to kernel crash" class="issue-link" data-issue-key="LU-7362"><del>LU-7362</del></a> lnet: Remove LASSERTS from router checker<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e6d6f7ab62877c5abc445734f7eaef4a81863796</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/17003/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17003/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7362" title="During our larger scale testing DVS was accidentally started on a router which could LNet to kernel crash" class="issue-link" data-issue-key="LU-7362"><del>LU-7362</del></a> lnet: Remove LASSERTS from router checker<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: df6cf859bbb29392064e6ddb701f3357e01b3a13</p> <p>Landed for 2.8</p> Related Development Rank 1|hzxrvr: Rank (Obsolete) 9223372036854775807 Severity