https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1114 Lustre <p>Example of affected thread :<br/> ============================<br/> PID: 7947 TASK: ffff881030721850 CPU: 1 COMMAND: "ptlrpcd_4"<br/> #0 <span class="error">&#91;ffff880044e27e90&#93;</span> crash_nmi_callback at ffffffff8101fd06<br/> 0000001 <span class="error">&#91;ffff880044e27ea0&#93;</span> notifier_call_chain at ffffffff814837f5<br/> 0000002 <span class="error">&#91;ffff880044e27ee0&#93;</span> atomic_notifier_call_chain at ffffffff8148385a<br/> 0000003 <span class="error">&#91;ffff880044e27ef0&#93;</span> notify_die at ffffffff8108026e<br/> 0000004 <span class="error">&#91;ffff880044e27f20&#93;</span> do_nmi at ffffffff81481443<br/> 0000005 <span class="error">&#91;ffff880044e27f50&#93;</span> nmi at ffffffff81480d50<br/> <span class="error">&#91;exception RIP: _spin_lock+30&#93;</span><br/> RIP: ffffffff8148062e RSP: ffff881030757da0 RFLAGS: 00000202<br/> RAX: 0000000000000000 RBX: ffff881030632540 RCX: 0000000000000000<br/> RDX: 0000000000000001 RSI: ffff88103c45e3d0 RDI: ffff88103c45e498<br/> RBP: ffff881030757da0 R8: ebc0de0100000000 R9: ffffffff00000100<br/> R10: 0000000000000000 R11: 000000000000000f R12: ffff881030632540<br/> R13: ffff881030632570 R14: ffff88103c45e3d0 R15: ffff88103c45e498<br/> ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018<br/> &#8212; &lt;NMI exception stack&gt; &#8212;<br/> 0000006 <span class="error">&#91;ffff881030757da0&#93;</span> _spin_lock at ffffffff8148062e<br/> 0000007 <span class="error">&#91;ffff881030757da8&#93;</span> ptlrpcd_check at ffffffffa05cfecc <span class="error">&#91;ptlrpc&#93;</span><br/> 0000008 <span class="error">&#91;ffff881030757e38&#93;</span> ptlrpcd at ffffffffa05d03ff <span class="error">&#91;ptlrpc&#93;</span><br/> 0000009 <span class="error">&#91;ffff881030757f48&#93;</span> kernel_thread at ffffffff810041aa<br/> ============================</p> <p>Concerned "partner" ptlrpcds-&gt;pd_threads[]-&gt;pc_lock spin_lock seems not initialized causing current ptlrpcd thread to spin for-ever !!!</p> <p>A possible fix for this problem should be to wait for all ptlrpcd/partner threads to fully initialize prior to start operations .... </p> LU-1114

ptrlrpd thread spinning since Lustre start on Client

Bug Major Resolved Duplicate nasf Alexandre Louvet Fri, 17 Feb 2012 13:03:57 +0000 Mon, 19 Nov 2012 17:14:38 +0000 Fri, 2 Mar 2012 06:56:03 +0000 Lustre 2.1.0 0 5 <p>Bruno</p> <p>Could you please confirm what version of the code you are running? You have marked this as a 2.1 issue but this seems to be related to mult-threaded ptlrpc which is a 2.2 feature</p> <p>Thanks</p> <p>Peter</p> <p>Bruno?</p> <p>In fact this is Lustre 2.1 + patch ORNL-22 applied.</p> <p>Fanyong</p> <p>Could you please comment on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Sorry for my silence Peter, but in-between others problems debugging, I was waiting for the confirmation from Bull R&amp;D of the infos/details that Aurelien added ...</p> <p>Concerning the scenario/conditions to trigger this issue we have no real nor interesting infos ...</p> <p>The only thing we can imagine/explain is that each ptlrpcd thread when starting, and according to PDB_POLICY_NODE, has only one partner choosen as "next core in the same NUMA node" but unfortunatelly this 2nd guy startup/init was not completed when the 1st guy tried to access the 2nd one's private data and to protect itself via the other's ptlrpcd_ctl.pc_lock which was found un-initialized, hence the dead-lock !!!</p> <p>As a possible fix, may be some kind of a "barrier" (with some additional+specific "started" flag) should be implemented to synchronize/ensure all ptlrpcd threads/partners start in ptlrpcd_init() ??...</p> <p>The PDB_POLICY_NODE is a ptlrpcd binding policy implemented by Bull that is NUMA aware.<br/> I have uploaded the patch into Gerrit <a href="http://review.whamcloud.com/2212" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2212</a> so the source code is available and can be integrated in the master.</p> <p>Note that there was a bug in the version running on the customer cluster.</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>"lustre/ptlrpc/ptlrpcd.c line 600"</b></div><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">for</span> (i = index+1; i != index; i = (i+1)%max) { </pre> </div></div> <p>which has been fixed in the uploaded patched:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">for</span> (i = (index+1)%max; i != index; i = (i+1)%max) { </pre> </div></div> <p>Gregoire</p> <p>Could you please resubmit your patch with <a href="https://jira.whamcloud.com/browse/LU-1114" title="ptrlrpd thread spinning since Lustre start on Client" class="issue-link" data-issue-key="LU-1114"><del>LU-1114</del></a> as the identified ticket number? We are trying to ensure that all landings to master have public LU tickets so that interested parties can read the full details relating to the change</p> <p>Thanks</p> <p>Peter</p> <p>I have created a separate ticket <a href="https://jira.whamcloud.com/browse/LU-1144" title="implement a NUMA aware ptlrpcd binding policy" class="issue-link" data-issue-key="LU-1144"><del>LU-1144</del></a> to track the implementation of the ptlrpcd binding policy PDB_POLICY_NODE.</p> <p>I have uploaded a patch that ensures partner thread control structure is accessed only when it is completely initialized: <a href="http://review.whamcloud.com/2227" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/2227</a>.</p> <p>The failure should not happend on original Lustre-2.2 branch if without other bind mode introduced. Because in original implementation, there is order control, when the partnership is established, private data for the partner should has been initialized already. For example: for the default bind mode "PDB_POLICY_PAIR", "&lt;0,1&gt;" are partners for each other, the partnership between "0" and "1" are established after "1"'s private data initialized. And according to ptlrpcd threads starting order, "0" should started before "1". So it can guarantee that: when &lt;0,1&gt; pair is established, both "0"'s and "1"'s private are ready.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> pc-&gt;pc_index = index; cfs_init_completion(&amp;pc-&gt;pc_starting); cfs_init_completion(&amp;pc-&gt;pc_finishing); cfs_spin_lock_init(&amp;pc-&gt;pc_lock); strncpy(pc-&gt;pc_name, name, sizeof(pc-&gt;pc_name) - 1); pc-&gt;pc_set = ptlrpc_prep_set(); <span class="code-keyword">if</span> (pc-&gt;pc_set == NULL) GOTO(out, rc = -ENOMEM); /* * So far only <span class="code-quote">"client"</span> ptlrpcd uses an environment. In the <span class="code-keyword">future</span>, * ptlrpcd thread (or a thread-set) has to be given an argument, * describing its <span class="code-quote">"scope"</span>. */ rc = lu_context_init(&amp;pc-&gt;pc_env.le_ctx, LCT_CL_THREAD|LCT_REMEMBER); <span class="code-keyword">if</span> (rc != 0) GOTO(out, rc); env = 1; #ifdef __KERNEL__ <span class="code-keyword">if</span> (index &gt;= 0) { <span class="code-comment">/* XXX: When <span class="code-quote">"1"</span> comes here, <span class="code-quote">"1"</span>'s <span class="code-keyword">private</span> data has been initialized, <span class="code-quote">"0"</span> is ready before <span class="code-quote">"1"</span> started. So here, we can establish the partnership between <span class="code-quote">"0"</span> and <span class="code-quote">"1"</span>.*/</span> rc = ptlrpcd_bind(index, max); <span class="code-keyword">if</span> (rc &lt; 0) GOTO(out, rc); } </pre> </div></div> <p>So consider my above comment, what's your thought? If it is introduced by the new bind mode "PDB_POLICY_NODE", do you think whether it is better to fix the issue inside such "PDB_POLICY_NODE" implementation patch?</p> <p>Thanks for looking.<br/> You are right the problem is specific to the PDB_POLICY_NODE which currently does not take care of establishing partnership only with ptlrpcd threads that are initialized.</p> <p>Therefore, I think this ticket can be closed (or marked duplicate of <a href="https://jira.whamcloud.com/browse/LU-1144" title="implement a NUMA aware ptlrpcd binding policy" class="issue-link" data-issue-key="LU-1144"><del>LU-1144</del></a>) and I will take this into account in the new version of the PDB_POLICY_NODE implementation I will post under <a href="https://jira.whamcloud.com/browse/LU-1144" title="implement a NUMA aware ptlrpcd binding policy" class="issue-link" data-issue-key="LU-1144"><del>LU-1144</del></a>.</p> <p>ok thanks Gregoire!</p> Development Rank 1|hzvhav: Rank (Obsolete) 6452 Severity