https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11981 Lustre <p>Over the span of about 20 minutes, routers reported the following in their console logs:<br/> 2019-02-19 10:05:02 <span class="error">&#91;330235.278414&#93;</span> LNetError: 33048:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (0, 2)<br/> 2019-02-19 10:05:02 <span class="error">&#91;330235.294305&#93;</span> LNetError: 33048:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 1646 previous similar messages</p> <p>While the lustre servers were being rebooted.<br/> (0, 2) corresponds to:<br/> msg-&gt;msg_ev.status == 0 (success)<br/> msg-&gt;msg_health_status == 2 (LNET_MSG_STATUS_LOCAL_DROPPED)</p> <p>See <a href="https://github.com/LLNL/lustre/releases" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/releases</a> for contents of 2.12.0_1.chaos.</p> clients and routers: Lustre 2.12.0_1.chaos <br/> lustre servers: Lustre 2.10.6_2.chaos <br/> <br/> Linux version 3.10.0-957.1.3.1chaos.ch6.x86_64 <br/> Clients OmniPath &lt;-&gt; routers &lt;-&gt; Servers mlx5 LU-11981

lnet_is_health_check() Msg is in inconsistent state, don't perform health checking (0, 2)

Bug Minor Resolved Fixed Amir Shehata Olaf Faaland llnl Wed, 20 Feb 2019 18:23:30 +0000 Thu, 30 Jan 2020 20:54:16 +0000 Fri, 20 Dec 2019 13:52:15 +0000 Lustre 2.12.0 Lustre 2.12.4 0 5 <p>would you be able to turn on net logging</p> <p>lctl set_param debug=+"net neterror"</p> <p>and capture the logs when you reproduce this message. I have made some changes in this area as part of <a href="https://jira.whamcloud.com/browse/LU-11477" title="handle health for both incoming and outgoing messages" class="issue-link" data-issue-key="LU-11477"><del>LU-11477</del></a>. I want to see if my changes there resolve this particular problem. I can then port it to 2.12.</p> <p>Thanks Amir. See dk.opal190.1550688817.txt.gz attached. Look towards the end of the file, the beginning starts before I turned on net logging.</p> <p>Sorry for the delay. It looks like there is a path in the code where the message is dropped but the message error status is not updated. However, the health status is updated. Leading to the inconsistent message you see. I'll update that path to correctly set the error status in the message</p> <p>&lt;poke&gt; Thanks</p> <p>there are a series of patches which were back ported to b2_12 which resolve some issues including the one reported in this ticket.</p> <p>The particular patch which resolves this issue is:</p> <p><a href="https://jira.whamcloud.com/browse/LU-12199" title="md&#39;s are not detached from uncommitted messages that have health check performed on them" class="issue-link" data-issue-key="LU-12199"><del>LU-12199</del></a> lnet: Ensure md is detached when msg is not committed</p> <p>However, I would suggest moving to 2.12.3 which includes this fix and others.</p> <p>Hmm.  We're seeing</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2019-12-11 10:01:08 [ 972.859958] LNetError: 28880:0:(lib-msg.c:820:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0)</pre> </div></div> <p>with Lustre 2.12.3.</p> <p>I see now that the values at the end of the message are (-125,0) which are different than the originally reported ones. And the system where I see this is Mellanox IB. New ticket for that?</p> <p>No need to create a new ticket. I would say this scenario is expected, since -125 is ECANCELED. For that we do not bother to adjust the health. The problem is that this message is at error level. We had already changed it to debug level, but that change was part of a bigger patch:</p> <p><a href="https://jira.whamcloud.com/browse/LU-11477" title="handle health for both incoming and outgoing messages" class="issue-link" data-issue-key="LU-11477"><del>LU-11477</del></a> lnet: handle health for incoming messages</p> <p>I'll create a patch just to change this log level to debug and push it on b2_12</p> <p>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37001" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37001</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11981" title="lnet_is_health_check() Msg is in inconsistent state, don&#39;t perform health checking (0, 2)" class="issue-link" data-issue-key="LU-11981"><del>LU-11981</del></a> lnet: clean up error message<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 96a278829b7a375e05f23b538f8db876a68caa71</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37001/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37001/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11981" title="lnet_is_health_check() Msg is in inconsistent state, don&#39;t perform health checking (0, 2)" class="issue-link" data-issue-key="LU-11981"><del>LU-11981</del></a> lnet: clean up error message<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: f549927ea633b910a8c788fa970af742b3bf10c1</p> <p>Landed for 2.12.4. Not needed on master</p> Related Development Rank 1|i00bxj: Rank (Obsolete) 9223372036854775807 Severity