https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5071 Lustre <p>Hello, </p> <p>We are seeing following error message on Lustre 2.5.1 clients, and it makes the system not responsive. multiple clients were affected with this issue.</p> <p>System Details: Lustre 2.5.1 / RHEL 6.5</p> <p>Here are the node names, time stamps and one according message: <br/> May 4 11:03:28 uc1n055 kernel: LustreError: 1979:0:(statahead.c:1704:do_statahead_enter()) ASSERTION( lli-&gt;u.d.d_sai == ((void *)0) ) failed: <br/> May 4 18:15:43 uc1n468 kernel: LustreError: 42888:0:(statahead.c:1704:do_statahead_enter()) ASSERTION( lli-&gt;u.d.d_sai == ((void *)0) ) failed: <br/> May 4 18:54:19 uc1n059 kernel: LustreError: 111650:0:(lovsub_lock.c:103:lovsub_lock_state()) ASSERTION( cl_lock_is_mutexed(slice-&gt;cls_lock) ) failed: <br/> May 9 09:21:08 uc1n129 kernel: LustreError: 93767:0:(statahead.c:1704:do_statahead_enter()) LBUG <br/> May 10 09:28:14 uc1n996 kernel: LustreError: 7387:0:(osc_lock.c:1224:osc_lock_wait()) LBUG <br/> May 15 07:50:57 uc1n198 kernel: LustreError: 25007:0:(statahead.c:1704:do_statahead_enter()) ASSERTION( lli-&gt;u.d.d_sai == ((void *)0) ) failed: </p> LU-5071

statahead.c:1704:do_statahead_enter()) ASSERTION( lli->u.d.d_sai == ((void *)0) ) failed:

Bug Major Resolved Fixed Zhenyu Xu Rajeshwaran Ganesan Fri, 16 May 2014 13:24:48 +0000 Sat, 6 Sep 2014 13:16:47 +0000 Tue, 2 Sep 2014 00:15:48 +0000 Lustre 2.5.1 0 6 <p>Rajesh</p> <p>Could you please confirm that it is vanilla 2.5.1 on both servers and clients for this cluster? Are any other Lustre versions or patches involved?</p> <p>Bobijam</p> <p>Does this seem related to existing tickets <a href="https://jira.whamcloud.com/browse/LU-4797" title="ASSERTION( cl_lock_is_mutexed(slice-&gt;cls_lock) ) failed" class="issue-link" data-issue-key="LU-4797"><del>LU-4797</del></a>/4693/4558?</p> <p>Thanks</p> <p>Peter</p> <p>Servers are in 2.4.3<br/> Clients are in 2.5.1</p> <p>the do_statahead_enter() LBUG can be cured by this back port patch <a href="http://review.whamcloud.com/10363" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/10363</a></p> <p>the lovsub_lock_state() LBUG was fixed in b2_5 branch, the patch is at <a href="http://review.whamcloud.com/9881" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9881</a></p> <p>Could you please provide source RPM with the patches?</p> <p>Rajesh</p> <p>These are included by default. For example, <a href="http://review.whamcloud.com/#/c/10363/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10363/</a> has a link to the build on the Jenkins server <a href="http://build.whamcloud.com/job/lustre-reviews/23961/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/23961/</a> Selecting the desired distro version allows you to drill into specific build artifacts - <a href="http://build.whamcloud.com/job/lustre-reviews/23961/arch=i686,build_type=server,distro=el6,ib_stack=inkernel/artifact/artifacts/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-reviews/23961/arch=i686,build_type=server,distro=el6,ib_stack=inkernel/artifact/artifacts/</a> , say. </p> <p>Peter</p> <p>Zhenyu has identified two of the LBUGs as <a href="https://jira.whamcloud.com/browse/LU-3498" title="most uses of IS_ERR_VALUE() are incorrect" class="issue-link" data-issue-key="LU-3498"><del>LU-3498</del></a> and <a href="https://jira.whamcloud.com/browse/LU-4558" title="Crash in cl_lock_put on racer" class="issue-link" data-issue-key="LU-4558"><del>LU-4558</del></a>, and both of those bugs are fixed in b2_5 and master. Since the LBUG which is in the summary of this ticket has been fixed, should this bug be resolved?</p> <p>I suppose there is still this LBUG:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 10 09:28:14 uc1n996 kernel: LustreError: 7387:0:(osc_lock.c:1224:osc_lock_wait()) LBUG </pre> </div></div> <p>But without any information other than the location of the LBUG, I think this bug isn't helpful. There is no information about that LBUG in any of the attachments either, as far as I can tell. If the bug will be kept open for the osc_lock_wait() LBUG, would it be possible to update the summary and description so that it doesn't look like <a href="https://jira.whamcloud.com/browse/LU-3498" title="most uses of IS_ERR_VALUE() are incorrect" class="issue-link" data-issue-key="LU-3498"><del>LU-3498</del></a>?</p> <p>In regards to the above comment...</p> <p>Does the above issue is fixed on 2.5.2, or still its a LBUG. Our customer saw the message once in the log. </p> <p>May 10 09:28:14 uc1n996 kernel: LustreError:7387:0:(osc_lock.c:1224:osc_lock_wait()) LBUG<br/> May 10 09:28:14 uc1n996 kernel: Pid: 7387, comm: less</p> <p>Not a lot of information here to go on. The assertion which was triggered looks like the same one as in <a href="https://jira.whamcloud.com/browse/LU-1356" title="Assertion triggered in osc_lock_wait()" class="issue-link" data-issue-key="LU-1356"><del>LU-1356</del></a>, but that bug was fixed way back in 2.3.0 and 2.1.4.</p> <p>Does it re-appears in 2.5.1 as well? </p> <p>ll_statahead_thread() calls ll_sai_get() at the first begining, but does not ll_sai_put() when ll_prep_md_op_data() failes. I think that might be the cause.</p> <p>Here is a patch which tries to fix this problem.<br/> <a href="http://review.whamcloud.com/#/c/10940/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10940/</a></p> <p>I think that the following assertion is already fixed by <a href="https://jira.whamcloud.com/browse/LU-3498" title="most uses of IS_ERR_VALUE() are incorrect" class="issue-link" data-issue-key="LU-3498"><del>LU-3498</del></a>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 4 11:03:28 uc1n055 kernel: LustreError: 1979:0:(statahead.c:1698:do_statahead_enter()) can't start ll_sa thread, rc: -2816 May 4 11:03:28 uc1n055 kernel: LustreError: 1979:0:(statahead.c:1704:do_statahead_enter()) ASSERTION( lli-&gt;u.d.d_sai == ((void *)0) ) failed: </pre> </div></div> <p>That assertion is inside an if block which is only executed when do_statahead_enter() <b>thinks</b> that the thread <b>creation</b> failed.</p> <p>Here is the relevant portion of do_statahead_enter() in version 2.5.1 (which has the <a href="https://jira.whamcloud.com/browse/LU-3498" title="most uses of IS_ERR_VALUE() are incorrect" class="issue-link" data-issue-key="LU-3498"><del>LU-3498</del></a> bug):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> rc = PTR_ERR(kthread_run(ll_statahead_thread, parent, <span class="code-quote">"ll_sa_%u"</span>, plli-&gt;lli_opendir_pid)); ... <span class="code-keyword">if</span> (IS_ERR_VALUE(rc)) { ... LASSERT(lli-&gt;lli_sai == NULL); RETURN(-EAGAIN); } </pre> </div></div> <p>So with the fix for <a href="https://jira.whamcloud.com/browse/LU-3498" title="most uses of IS_ERR_VALUE() are incorrect" class="issue-link" data-issue-key="LU-3498"><del>LU-3498</del></a>, this code will not be executed unless the thread creation actually fails. If the thread creation fails, your patched code which does an extra ll_sai_put(sai) will not be executed anyway.</p> <p>Unless I'm missing something, I don't think this patch belongs to this ticket. It probably is more appropriate to link that patch against <a href="https://jira.whamcloud.com/browse/LU-5274" title="ll_statahead_thread() may leak parent, uses parent after dput" class="issue-link" data-issue-key="LU-5274"><del>LU-5274</del></a>.</p> <p>2.5.1 code has a glitch, kthread_run() returns thread id which could be a big value, and IS_ERR defines (-1000, -1) which is too narrow. 2.5.3 code does not has this issue.</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>2.5.1</b></div><div class="codeContent panelContent"> <pre class="code-java">#define IS_ERR(a) ((unsigned <span class="code-object">long</span>)(a) &gt; (unsigned <span class="code-object">long</span>)-1000L) </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>2.5.3</b></div><div class="codeContent panelContent"> <pre class="code-java"># define IS_ERR_VALUE(x) ((x) &gt;= (unsigned <span class="code-object">long</span>)-4095) </pre> </div></div> Related LU-4558 LU-3498 LU-5274 Development Rank 1|hzwmmv: Rank (Obsolete) 14002 Severity