https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7351 Lustre <p>In our testing on our medium size Cray system we encountered the following crash while attempting to bring up LNet on the routers.</p> <p><span class="error">&#91;2015-10-26 15:51:14&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Lustre: kgnilnd build version: 2.7.61-DNE2-1.0502.0.2.7-jsimmons-Unknown-2015-10-21-11:16<br/> <span class="error">&#91;2015-10-26 15:51:14&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>LNet: Added LNI 12@gni2 <span class="error">&#91;16/8192/0/0&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>LNetError: 149:0:(o2iblnd_cb.c:2239:kiblnd_passive_connect()) Can't accept conn from 10.36.226.4@o2ib on NA (ib0:0:10.36.223.1): bad dst nid 10.36.223.1@o2ib<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>BUG: unable to handle kernel NULL pointer dereference at 0000000000000080<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>IP: <span class="error">&#91;&lt;ffffffffa03e3e5b&gt;&#93;</span> kiblnd_passive_connect+0xfb/0x16b0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>PGD 3dc9ae067 PUD 3ddc0c067 PMD 0 <br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Oops: 0000 <a href="#1" target="_blank" rel="noopener">1</a> SMP <br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>CPU 5 <br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Modules linked in: ko2iblnd kgnilnd lnet crc32c libcfs binfmt_misc rdma_ucm ib_ucm rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_uverbs ib_umad mlx5_ib mlx5_core mlx4_en mlx4_ib ib_sa ib_mad ib_core mlx4_core compat nic_compat dm_mod kdreg gpcd_gem ipogif_gem kgni_gem hwerr(P) rca hss_os(P) heartbeat simplex(P) ghal_gem cgm craytrace<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Pid: 149, comm: kworker/5:1 Tainted: P 3.0.101-0.46.1_1.0502.8871-cray_gem_s #1 <br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>RIP: 0010:<span class="error">&#91;&lt;ffffffffa03e3e5b&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa03e3e5b&gt;&#93;</span> kiblnd_passive_connect+0xfb/0x16b0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>RSP: 0018:ffff8803f1b0fb10 EFLAGS: 00010246<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>RAX: 000000000000003f RBX: ffffffffa03ee513 RCX: ffffffff81368c50<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>RDX: 0000000000000100 RSI: 0000000000000000 RDI: ffff8803e8ac7680<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>RBP: ffff8803f1b0fbd0 R08: 0000000000000005 R09: 0000000000000005<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>R10: 0000000000000003 R11: 00000000ffffffff R12: 0000000000000012<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>R13: 0000000000000000 R14: 0000000000000000 R15: ffff8803c0e58620<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>FS: 00007f9e343b7700(0000) GS:ffff880407d40000(0000) knlGS:0000000000000000<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>CR2: 0000000000000080 CR3: 00000003dc9bd000 CR4: 00000000000007e0<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>LNet: Added LNI 10.36.223.1@o2ib <span class="error">&#91;63/2560/0/180&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Process kworker/5:1 (pid: 149, threadinfo ffff8803f1b0c000, task ffff8803f1b09040)<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Stack:<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> ffff8803c0e58620 ffffffffa02f6080 0000000000000000 ffff8803ea8bcc00<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> ffffffffa02f6080 000500000a24e204 0000000000000001 ffff8803e4740000<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> 000300120be91b91 0000000000000000 000010000000003f ffffffffa017b636<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Call Trace:<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa03e59bd&gt;&#93;</span> kiblnd_cm_callback+0x5ad/0x2070 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa024145b&gt;&#93;</span> cma_req_handler+0x1eb/0x550 <span class="error">&#91;rdma_cm&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa016ff57&gt;&#93;</span> cm_process_work+0x27/0x130 <span class="error">&#91;ib_cm&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa0171fb0&gt;&#93;</span> cm_req_handler+0x750/0xa00 <span class="error">&#91;ib_cm&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa0172385&gt;&#93;</span> cm_work_handler+0x125/0xf4c <span class="error">&#91;ib_cm&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81060953&gt;&#93;</span> process_one_work+0x163/0x440<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81063473&gt;&#93;</span> worker_thread+0x183/0x400<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81067ace&gt;&#93;</span> kthread+0x9e/0xb0<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81490074&gt;&#93;</span> kernel_thread_helper+0x4/0x10<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Code: 0f 84 da 01 00 00 66 3d 00 11 0f 84 d0 01 00 00 66 c7 45 84 12 00 45 31 f6 48 8b 05 38 03 01 00 ba 00 01 00 00 8b 00 66 89 45 90 <br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> 8b 86 80 00 00 00 85 c0 0f 45 d0 48 8b bd 58 ff ff ff 48 8d <br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>RIP <span class="error">&#91;&lt;ffffffffa03e3e5b&gt;&#93;</span> kiblnd_passive_connect+0xfb/0x16b0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> RSP &lt;ffff8803f1b0fb10&gt;<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>CR2: 0000000000000080<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>--<del>[ end trace 311d9fd8dd61b1cf ]</del>--<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Kernel panic - not syncing: Fatal exception<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Pid: 149, comm: kworker/5:1 Tainted: P D 3.0.101-0.46.1_1.0502.8871-cray_gem_s #1<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span>Call Trace:<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81006651&gt;&#93;</span> try_stack_unwind+0x161/0x1a0<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81004eb9&gt;&#93;</span> dump_trace+0x89/0x430<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff810060bc&gt;&#93;</span> show_trace_log_lvl+0x5c/0x80<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff810060f5&gt;&#93;</span> show_trace+0x15/0x20<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff8148b31c&gt;&#93;</span> dump_stack+0x79/0x84<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff8148b3bb&gt;&#93;</span> panic+0x94/0x1da<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81005ed8&gt;&#93;</span> oops_end+0xa8/0xe0<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81027589&gt;&#93;</span> no_context+0xf9/0x260<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81027855&gt;&#93;</span> __bad_area_nosemaphore+0x165/0x1f0<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff810278f3&gt;&#93;</span> bad_area_nosemaphore+0x13/0x20<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81027e4e&gt;&#93;</span> do_page_fault+0x2fe/0x440<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff8148e8cf&gt;&#93;</span> page_fault+0x1f/0x30<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa03e3e5b&gt;&#93;</span> kiblnd_passive_connect+0xfb/0x16b0 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa03e59bd&gt;&#93;</span> kiblnd_cm_callback+0x5ad/0x2070 <span class="error">&#91;ko2iblnd&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa024145b&gt;&#93;</span> cma_req_handler+0x1eb/0x550 <span class="error">&#91;rdma_cm&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa016ff57&gt;&#93;</span> cm_process_work+0x27/0x130 <span class="error">&#91;ib_cm&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa0171fb0&gt;&#93;</span> cm_req_handler+0x750/0xa00 <span class="error">&#91;ib_cm&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffffa0172385&gt;&#93;</span> cm_work_handler+0x125/0xf4c <span class="error">&#91;ib_cm&#93;</span><br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81060953&gt;&#93;</span> process_one_work+0x163/0x440<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81063473&gt;&#93;</span> worker_thread+0x183/0x400<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81067ace&gt;&#93;</span> kthread+0x9e/0xb0<br/> <span class="error">&#91;2015-10-26 15:51:15&#93;</span><span class="error">&#91;c0-0c0s6n0&#93;</span> <span class="error">&#91;&lt;ffffffff81490074&gt;&#93;</span> kernel_thread_helper+0x4/0x10</p> Cray routers running Lustre 2.7.61 in an SLES11 SP3 environment. LU-7351

LNet router crash during bring up of infiniband interface.

Bug Blocker Resolved Duplicate Doug Oucharek James A Simmons Wed, 28 Oct 2015 17:43:53 +0000 Fri, 20 Nov 2015 20:55:26 +0000 Fri, 20 Nov 2015 18:45:51 +0000 Lustre 2.8.0 Lustre 2.8.0 0 11 <p>James, what IB card is in the router (FDR, EDR)? Looks like mlx5 is being used. Is this upstream OFED or MOFED?</p> <p>Its a mlx5 FDR card using the OFED 3.12 stack.</p> <p>Is this a production system or a test system? Should it be sev 1 (highest priority)?</p> <p>This was tested on production system. We had to roll back to 2.5 version to have it work again. I made it a blocker since it prevents Lustre bring up for anyone attempting to the latest pre-2.8 clients.</p> <p>I agree this is a blocker. Adding sev 1 means a production system is down and needs immediate attention to get it back up again. If the system in question is back up and running, can we change this to a sev 2?</p> <p>Sure you can change it to 2.</p> <p>The o2iblnd_cb.c I have in 2.7.61 does not seem to match yours. Can you attach your copy of o2iblnd_cb.c so I can see the differences?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">(gdb) l *kiblnd_cm_callback+0x5ad 0x1661d is in kiblnd_cm_callback (/home/ashehata/lustre-master/lnet/klnds/o2iblnd/o2iblnd_cb.c:2889). 2884 kiblnd_peer_decref(peer); 2885 <span class="code-keyword">return</span> rc; <span class="code-comment">/* rc != 0 destroys cmid */</span> 2886 2887 <span class="code-keyword">case</span> RDMA_CM_EVENT_ROUTE_ERROR: 2888 peer = (kib_peer_t *)cmid-&gt;context; 2889 CNETERR(<span class="code-quote">"%s: ROUTE ERROR %d\n"</span>, 2890 libcfs_nid2str(peer-&gt;ibp_nid), event-&gt;status); 2891 kiblnd_peer_connect_failed(peer, 1, -EHOSTUNREACH); 2892 kiblnd_peer_decref(peer); 2893 <span class="code-keyword">return</span> -EHOSTUNREACH; <span class="code-comment">/* rc != 0 destroys cmid */</span> </pre> </div></div> <p>looks like peer might be NULL<br/> If there is a change in the way OFED works that might explain this error.<br/> can you put a few debug statements around this part to see if this is indeed the case? A check to see if cmid or peer are NULL.</p> <p>Amir: the previous line is a call to kiblnd_peer_connect_failed() where peer is dereferenced. I would have thought the crash would have happened there. Are you looking at the proper binary? The files James sent us are different than 2.7.61.</p> <p>James: we suspect that cmid-&gt;context is coming back NULL when we don't expect such a thing to happen. Can you verify the line of the crash with your binary? I'm not sure that the binary Amir used is equivalent to yours. Once we know for sure that a NULL cmid-&gt;context is the cause, we can start to figure out how such a thing can happen.</p> <p>This sounds like it is a blocker for the 2.8 release as stated. I marked it as such, but you can correct it if I'm wrong.</p> <p>Nope the problem is not cmd-&gt;context being NULL. I'm going to give it another run to see what it is.</p> <p>I added the latest <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> patch and I'm not seeing the crashes anymore. This is at a smallest scale that I saw this problem before but we don't know if it resolves this at titan scales.</p> <p>James: Can I close this linked to <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a>?</p> <p>Can we wait until <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> is settled. I noticed that patch has changed again but is now disliked. I want to wait until a agreed on solution is presented before I will try to test it again. Is that okay?</p> <p>Hi James. I see you successfully tested <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a>. Does that mean this issue has been resolved?</p> <p>Seems like this is a duplicate <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a>. James, please speak up if you think otherwise</p> <p>I agree. The patch from <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> resolves this.</p> <p>Great - thanks for confirming James</p> Related LU-3322 Development Epic/Theme lnet Rank 1|hzxrpb: Rank (Obsolete) 9223372036854775807 Severity