https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5151 Lustre <p>While testing 2.6 in my Cray test environment I keep losing my routers which NMI produces the following back traces:</p> <p>2014-06-05T16:45:09.828951-04:00 c0-0c0s2n3 Pid: 4554, comm: kiblnd_sd_01_01 Tainted: P N 3.0.82-0.7.9_1.0502.7780-cray_gem_s #1 <br/> 2014-06-05T16:45:09.828965-04:00 c0-0c0s2n3 RIP: 0010:<span class="error">&#91;&lt;ffffffffa0341831&gt;&#93;</span> <span class="error">&#91;&lt;ffffffffa0341831&gt;&#93;</span> lnet_return_rx_credits_locked+0x171/0x310 <span class="error">&#91;lnet&#93;</span><br/> 2014-06-05T16:45:09.828971-04:00 c0-0c0s2n3 RSP: 0018:ffff8803ea379bb0 EFLAGS: 00010286<br/> 2014-06-05T16:45:09.858936-04:00 c0-0c0s2n3 RAX: dead000000200200 RBX: ffff880317d5a800 RCX: 00000000ffffffff<br/> 2014-06-05T16:45:09.858949-04:00 c0-0c0s2n3 RDX: dead000000100100 RSI: 0000000000000001 RDI: ffff880317d5a800<br/> 2014-06-05T16:45:09.858960-04:00 c0-0c0s2n3 RBP: ffff8803ea379be0 R08: ffff8803e821c860 R09: ffff880317d5a850<br/> 2014-06-05T16:45:09.858970-04:00 c0-0c0s2n3 R10: 0000000000000000 R11: 0000000000000000 R12: ffff880317d5a800<br/> 2014-06-05T16:45:09.858977-04:00 c0-0c0s2n3 R13: ffff8803daf91880 R14: 00000000fffffff5 R15: 0000000000000001<br/> 2014-06-05T16:45:09.888794-04:00 c0-0c0s2n3 FS: 00007f28c44457a0(0000) GS:ffff880407cc0000(0000) knlGS:0000000000000000<br/> 2014-06-05T16:45:09.888807-04:00 c0-0c0s2n3 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br/> 2014-06-05T16:45:09.888818-04:00 c0-0c0s2n3 CR2: 000000000063c800 CR3: 000000031f33f000 CR4: 00000000000007e0<br/> 2014-06-05T16:45:09.888824-04:00 c0-0c0s2n3 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br/> 2014-06-05T16:45:09.888834-04:00 c0-0c0s2n3 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br/> 2014-06-05T16:45:09.918910-04:00 c0-0c0s2n3 Process kiblnd_sd_01_01 (pid: 4554, threadinfo ffff8803ea378000, task ffff8803e89480c0)<br/> 2014-06-05T16:45:09.918924-04:00 c0-0c0s2n3 Stack:<br/> 2014-06-05T16:45:09.918940-04:00 c0-0c0s2n3 ffff8803ea379bd0 ffff880317d5a800 0000000000000001 0000000000000001<br/> 2014-06-05T16:45:09.918951-04:00 c0-0c0s2n3 00000000fffffff5 0000000000000001 ffff8803ea379c10 ffffffffa0338b28<br/> 2014-06-05T16:45:09.918956-04:00 c0-0c0s2n3 ffff880317d5a918 dead000000200200 ffff880317d5a800 ffff8803e9b18d80<br/> 2014-06-05T16:45:09.918961-04:00 c0-0c0s2n3 Call Trace:<br/> 2014-06-05T16:45:09.918966-04:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa0338b28&gt;&#93;</span> lnet_msg_decommit+0xf8/0x6b0 <span class="error">&#91;lnet&#93;</span><br/> 2014-06-05T16:45:09.948770-04:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa0339b47&gt;&#93;</span> lnet_finalize+0x297/0x7d0 <span class="error">&#91;lnet&#93;</span><br/> 2014-06-05T16:45:09.948783-04:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03465ed&gt;&#93;</span> lnet_parse+0xc2d/0x1b80 <span class="error">&#91;lnet&#93;</span><br/> 2014-06-05T16:45:09.948794-04:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03db68a&gt;&#93;</span> kiblnd_handle_rx+0x30a/0x690 <span class="error">&#91;ko2iblnd&#93;</span><br/> 2014-06-05T16:45:09.948805-04:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03e03af&gt;&#93;</span> kiblnd_rx_complete+0x34f/0x420 <span class="error">&#91;ko2iblnd&#93;</span><br/> 2014-06-05T16:45:09.948815-04:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffffa03e0d25&gt;&#93;</span> kiblnd_scheduler+0x7c5/0x970 <span class="error">&#91;ko2iblnd&#93;</span><br/> 2014-06-05T16:45:09.948821-04:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff810672fe&gt;&#93;</span> kthread+0x9e/0xb0<br/> 2014-06-05T16:45:09.978765-04:00 c0-0c0s2n3 <span class="error">&#91;&lt;ffffffff81481874&gt;&#93;</span> kernel_thread_helper+0x4/0x10<br/> 2014-06-05T16:45:09.978785-04:00 c0-0c0s2n3 Code: c2 0f 85 2b 01 00 00 8d 41 01 85 c0 41 89 45 48 0f 8f dc fe ff ff 49 8b 7d 20 be 01 00 00 00 48 83 <br/> ef 10 48 8b 47 18 48 8b 57 10 <br/> 2014-06-05T16:45:10.004304-04:00 c0-0c0s2n3 89 42 08 48 89 10 48 b8 00 01 10 00 00 00 ad de 48 89 47 10 <br/> 2014-06-05T16:45:10.004326-04:00 c0-0c0s2n3 RIP <span class="error">&#91;&lt;ffffffffa0341831&gt;&#93;</span> lnet_return_rx_credits_locked+0x171/0x310 <span class="error">&#91;lnet&#93;</span><br/> 2014-06-05T16:45:10.004333-04:00 c0-0c0s2n3 RSP &lt;ffff8803ea379bb0&gt;<br/> 2014-06-05T16:45:10.029888-04:00 c0-0c0s2n3 --<del>[ end trace 17126666cf42dece ]</del>--</p> Cray router to connect infiniband to gemini interconnect. LU-5151

Oops in lnet_return_rx_credits_locked

Bug Blocker Resolved Fixed Liang Zhen James A Simmons lnet Thu, 5 Jun 2014 22:24:47 +0000 Wed, 11 Jun 2014 13:41:54 +0000 Wed, 11 Jun 2014 13:41:54 +0000 Lustre 2.6.0 Lustre 2.6.0 0 8 <p>Liang, </p> <p>Would you please comment on this ticket?</p> <p>Thank you, <br/> James</p> <p>Hi James, to narrow down the problem, have you ever seen this issue with other versions between 2.4 and 2.6? thanks</p> <p>I see the reason here, I think it's because this patch changed returned value of lnet_post_routed_recv_locked() from positive to negative (<a href="http://review.whamcloud.com/#/c/9369/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9369/</a>)<br/> which means lnet_parse_forward_locked()-&gt;lnet_post_routed_recv_locked() will return -EAGAIN instead of EAGAIN, and it will be treated as real error in lnet_parse, although it is not a real error because EAGAIN means message is waiting for router buffer, and we will try to finalise a message which is still queued:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> if (!for_me) { rc = lnet_parse_forward_locked(ni, msg); lnet_net_unlock(cpt); if (rc &lt; 0) goto free_drop; if (rc == 0) { lnet_ni_recv(ni, msg-&gt;msg_private, msg, 0, 0, payload_length, payload_length); } return 0; } </pre> </div></div> <p>I have only seen the problem with 2.6.</p> <p>patch is here: <a href="http://review.whamcloud.com/#/c/10625/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10625/</a></p> <p>I changed it from EAGAIN to -EAGAIN so it matches the behavior in the upstream kernel. It is frowned on to use EAGAIN in kernel space. How about instead we just do:</p> <p>diff --git a/lnet/lnet/lib-move.c b/lnet/lnet/lib-move.c<br/> index 6097ae0..5fcc19b 100644<br/> &#8212; a/lnet/lnet/lib-move.c<br/> +++ b/lnet/lnet/lib-move.c<br/> @@ -1961,12 +1961,14 @@ lnet_parse(lnet_ni_t *ni, lnet_hdr_t *hdr, lnet_nid_t from_nid,<br/> rc = lnet_parse_forward_locked(ni, msg);<br/> lnet_net_unlock(cpt);</p> <ul class="alternate" type="square"> <li>if (rc &lt; 0)<br/> + if (rc == -EAGAIN) /* waiting for buffer */<br/> + return 0;<br/> +<br/> + if (rc != 0)<br/> goto free_drop;</li> <li>if (rc == 0) { - lnet_ni_recv(ni, msg-&gt;msg_private, msg, 0, - 0, payload_length, payload_length); - } <p>+<br/> + lnet_ni_recv(ni, msg-&gt;msg_private, msg, 0,<br/> + 0, payload_length, payload_length);<br/> return 0;<br/> }</p></li> </ul> <p>I tired the above version of my patch and it resolved the issue. Do you Liang mind if we go with that version instead.</p> <p>Hi James, both way should be fine, my concern about for -EAGAIN is, we need to impose another condition to LND: -EAGAIN should never returned by lnet_ni_eager_recv.<br/> It is true for now, but it's probably better not to have this limit?</p> <p>I wouldn't consider that a huge limitation. The LASSERT in lnet_ni_eager_recv would handle this case today. So a potential driver writer would know not to return a -EAGAIN.</p> <p>I see their are strong opinions on this. I purpose that if you want to continue positive values in some of the function that we call them something else besides EAGAIN and ENOENT. This will be flagged by the HPPD checker and upstream using those values will be frowned on. So I suggest you define your own errors so EAGAIN will become LNET_RETRY and ENOENT becomes LNET_MISMATCH. Can you live with this compromise?</p> <p>sorry I didn't notice there is update and review comments on patch, and overwrote it. <br/> After rethink, I agree with Isaac and still tend to keep positive value, it will be much cleaner to me.</p> <p>James, thanks for reporting this problem. We have seen this at Cray as well, of course, and only on routers as you had. We have been working around the problem by using b2_5 vintage routers.</p> <p>Thank you Liang for working with me on this issue. IMHO the new LNET_CREDIT_* values make it far more clear what is going on than using EAGAIN or just returning zero.</p> <p>welcome, it's assigned to me anyway <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/>. I think we can close it now because it's landed which is the only place we have this issue.</p> <p>Patch landed to Master.</p> Development Epic lnet Rank 1|hzwnuv: Rank (Obsolete) 14214 Severity