https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2272 Lustre <p>Running racer with a debugging kernel on master I hit this condition:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 8529.226634] BUG: spinlock cpu recursion on CPU#2, ll_sa_30972/30992 (Not tainted) [ 8529.227125] lock: ffff88009cb38eb8, .magic: dead4ead, .owner: ptlrpcd_2/7457, .owner_cpu: 2 [ 8529.227595] Pid: 30992, comm: ll_sa_30972 Not tainted 2.6.32-debug #6 [ 8529.227874] Call Trace: [ 8529.228084] [&lt;ffffffff8128098a&gt;] ? spin_bug+0xaa/0x100 [ 8529.228342] [&lt;ffffffff81280ba1&gt;] ? _raw_spin_lock+0x121/0x180 [ 8529.228610] [&lt;ffffffff814fafde&gt;] ? _spin_lock+0xe/0x10 [ 8529.228892] [&lt;ffffffffa0d3192c&gt;] ? do_statahead_interpret+0x4c/0xdd0 [lustre] [ 8529.229426] [&lt;ffffffff8109011c&gt;] ? remove_wait_queue+0x3c/0x50 [ 8529.229718] [&lt;ffffffffa0d3690a&gt;] ? ll_statahead_thread+0xcda/0xf40 [lustre] [ 8529.230008] [&lt;ffffffff81057d60&gt;] ? default_wake_function+0x0/0x20 [ 8529.230293] [&lt;ffffffffa0d35c30&gt;] ? ll_statahead_thread+0x0/0xf40 [lustre] [ 8529.230577] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [ 8529.230844] [&lt;ffffffffa0d35c30&gt;] ? ll_statahead_thread+0x0/0xf40 [lustre] [ 8529.231142] [&lt;ffffffffa0d35c30&gt;] ? ll_statahead_thread+0x0/0xf40 [lustre] [ 8529.231429] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> LU-2272

BUG: spinlock cpu recursion on CPU#2, ll_sa_30972/30992

Bug Critical Resolved Duplicate Lai Siyao Oleg Drokin Sat, 3 Nov 2012 20:28:48 +0000 Fri, 11 Dec 2015 15:49:07 +0000 Sun, 22 Sep 2013 13:47:21 +0000 Lustre 2.4.0 Lustre 2.7.0 0 9 <p>Seems to be related to statahead, so FanYong, can you please take a look?</p> <p>OK, try to reproduce by myself and dump the log.</p> <p>I just had this hit again in my racer testing on veryfresh master:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Dec 11 15:15:42 centos6-8 kernel: [226594.953228] BUG: spinlock cpu recursion on CPU#0, ll_sa_13669/13959 (Not tainted) Dec 11 15:15:42 centos6-8 kernel: [226594.953702] lock: ffff880018c19eb8, .magic: dead4ead, .owner: ptlrpcd_2/26580, .owner_cpu: 0 Dec 11 15:15:42 centos6-8 kernel: [226594.954248] Pid: 13959, comm: ll_sa_13669 Not tainted 2.6.32-debug #6 Dec 11 15:15:42 centos6-8 kernel: [226594.954815] Call Trace: Dec 11 15:15:42 centos6-8 kernel: [226594.955034] [&lt;ffffffff8128098a&gt;] ? spin_bug+0xaa/0x100 Dec 11 15:15:42 centos6-8 kernel: [226594.955285] [&lt;ffffffff81280ba1&gt;] ? _raw_spin_lock+0x121/0x180 Dec 11 15:15:42 centos6-8 kernel: [226594.955548] [&lt;ffffffff814fafde&gt;] ? _spin_lock+0xe/0x10 Dec 11 15:15:42 centos6-8 kernel: [226594.955851] [&lt;ffffffffa0e368dc&gt;] ? do_statahead_interpret+0x4c/0xdd0 [lustre] Dec 11 15:15:42 centos6-8 kernel: [226594.957300] [&lt;ffffffff8109011c&gt;] ? remove_wait_queue+0x3c/0x50 Dec 11 15:15:42 centos6-8 kernel: [226594.957612] [&lt;ffffffffa0e3b90a&gt;] ? ll_statahead_thread+0xcda/0xf40 [lustre] Dec 11 15:15:42 centos6-8 kernel: [226594.957977] [&lt;ffffffff81057d60&gt;] ? default_wake_function+0x0/0x20 Dec 11 15:15:42 centos6-8 kernel: [226594.958317] [&lt;ffffffffa0e3ac30&gt;] ? ll_statahead_thread+0x0/0xf40 [lustre] Dec 11 15:15:42 centos6-8 kernel: [226594.959502] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>Ok, I finally got to the root of it.<br/> The reason we got it is in ll_statahead_interpret we sleep under spinlock:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; bt 21485 PID: 21485 TASK: ffff8800b2906540 CPU: 7 COMMAND: "ptlrpcd_4" #0 [ffff8800b098f930] schedule at ffffffff814fae3a #1 [ffff8800b098f9f8] __cond_resched at ffffffff810644ea #2 [ffff8800b098fa18] _cond_resched at ffffffff814fb840 #3 [ffff8800b098fa28] __kmalloc at ffffffff811686d0 #4 [ffff8800b098fa78] cfs_alloc at ffffffffa0b0cb90 [libcfs] #5 [ffff8800b098faa8] ldlm_bl_to_thread at ffffffffa1268cc1 [ptlrpc] #6 [ffff8800b098fbb8] ldlm_bl_to_thread_lock at ffffffffa1269219 [ptlrpc] #7 [ffff8800b098fbc8] ldlm_lock_decref_internal at ffffffffa1246ccd [ptlrpc] #8 [ffff8800b098fc28] ldlm_lock_decref at ffffffffa1247d69 [ptlrpc] #9 [ffff8800b098fc58] ll_intent_drop_lock at ffffffffa07dac8d [lustre] #10 [ffff8800b098fc88] ll_statahead_interpret at ffffffffa0838e66 [lustre] #11 [ffff8800b098fce8] mdc_intent_getattr_async_interpret at ffffffffa0dea4a2 [mdc] #12 [ffff8800b098fd68] ptlrpc_check_set at ffffffffa12804e2 [ptlrpc] #13 [ffff8800b098fe08] ptlrpcd_check at ffffffffa12adc5b [ptlrpc] #14 [ffff8800b098fe68] ptlrpcd at ffffffffa12ae1a3 [ptlrpc] #15 [ffff8800b098ff48] kernel_thread at ffffffff8100c10a </pre> </div></div> <p>While I mostly hit the warnings like in this ticket, right now I got a deadlock, and that's how I finally found this issue.<br/> offending code:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> entry-&gt;se_minfo = minfo; entry-&gt;se_req = ptlrpc_request_addref(req); /* Release the async ibits lock ASAP to avoid deadlock * when statahead thread tries to enqueue lock on parent * for readpage and other tries to enqueue lock on child * with parent's lock held, for example: unlink. */ entry-&gt;se_handle = it-&gt;d.lustre.it_lock_handle; can sleep!!! =&gt; ll_intent_drop_lock(it); wakeup = sa_received_empty(sai); cfs_list_add_tail(&amp;entry-&gt;se_list, &amp;sai-&gt;sai_entries_received); } sai-&gt;sai_replied++; spin_unlock(&amp;lli-&gt;lli_sa_lock); </pre> </div></div> <p>Lai</p> <p>Are you able to advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>The fix will modify the same code of patch for <a href="https://jira.whamcloud.com/browse/LU-3270" title="ptlrpcd strnlen crash trying to log a message" class="issue-link" data-issue-key="LU-3270"><del>LU-3270</del></a>, so I based this fix on <a href="https://jira.whamcloud.com/browse/LU-3270" title="ptlrpcd strnlen crash trying to log a message" class="issue-link" data-issue-key="LU-3270"><del>LU-3270</del></a>: <a href="http://review.whamcloud.com/#change,6413" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6413</a></p> <p>The above patch is abandoned, and the fix is included in <a href="http://review.whamcloud.com/#/c/6392/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/6392/</a>.</p> <p>Lai</p> <p>So can we duplicate this ticket into <a href="https://jira.whamcloud.com/browse/LU-3270" title="ptlrpcd strnlen crash trying to log a message" class="issue-link" data-issue-key="LU-3270"><del>LU-3270</del></a>?</p> <p>Peter</p> <p>Yes, Peter. <a href="https://jira.whamcloud.com/browse/LU-3270" title="ptlrpcd strnlen crash trying to log a message" class="issue-link" data-issue-key="LU-3270"><del>LU-3270</del></a> fixed a number of statahead bugs which including this one.</p> <p>ok thanks!</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13846" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13846</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-2272" title="BUG: spinlock cpu recursion on CPU#2, ll_sa_30972/30992" class="issue-link" data-issue-key="LU-2272"><del>LU-2272</del></a> statahead: ll_intent_drop_lock() called in spinlock<br/> Project: fs/lustre-release<br/> Branch: b2_5<br/> Current Patch Set: 1<br/> Commit: 668bbb377ab45b9e863844406682c865de684b66</p> Related Development Rank 1|hzvbnj: Rank (Obsolete) 5431 Severity