https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5052 Lustre <p>MDS build up high load with no cpu activity. Lustre dumping call trace to console. (looks like dup of <a href="https://jira.whamcloud.com/browse/LU-4794" title="MDS threads all stuck in jbd2_journal_start" class="issue-link" data-issue-key="LU-4794"><del>LU-4794</del></a>. If so please advise when the patch will land) </p> <p>Attached is full stack trace for all threads.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">INFO: task ldlm_cn_00:6299 blocked <span class="code-keyword">for</span> more than 120 seconds. <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. ldlm_cn_00 D 000000000000001a 0 6299 2 0x00000080 ffff881ec525db30 0000000000000046 0000000000000000 ffffffff8129507e ffff881ec525dad0 00000000dcd2dc2e ffff881fb0bd8d00 ffff881ec525dad0 ffff881fafe73098 ffff881ec525dfd8 000000000000fc40 ffff881fafe73098 Call Trace: [&lt;ffffffff8129507e&gt;] ? number+0x2ee/0x320 [&lt;ffffffffa055c14a&gt;] start_this_handle+0x27a/0x4a0 [jbd2] [&lt;ffffffff8108ff00&gt;] ? autoremove_wake_function+0x0/0x40 [&lt;ffffffffa055c570&gt;] jbd2_journal_start+0xd0/0x110 [jbd2] [&lt;ffffffffa08e6338&gt;] ldiskfs_journal_start_sb+0x58/0x90 [ldiskfs] [&lt;ffffffffa072c017&gt;] fsfilt_ldiskfs_start+0x77/0x5e0 [fsfilt_ldiskfs] [&lt;ffffffffa07a9ac0&gt;] llog_origin_handle_cancel+0x4b0/0xd70 [ptlrpc] [&lt;ffffffffa076f71f&gt;] ldlm_cancel_handler+0x1bf/0x5e0 [ptlrpc] [&lt;ffffffffa079fb4e&gt;] ptlrpc_main+0xc4e/0x1a40 [ptlrpc] [&lt;ffffffffa079ef00&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffffa079ef00&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] [&lt;ffffffffa079ef00&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 INFO: task ldlm_cb_00:6302 blocked <span class="code-keyword">for</span> more than 120 seconds. <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. ldlm_cb_00 D 0000000000000002 0 6302 2 0x00000080 ffff881ec5265b20 0000000000000046 0000000000000000 000000ab00000000 ffff881ec5265b50 ffffffff8129507e 3634333236363330 3134363536363336 ffff881ec5263af8 ffff881ec5265fd8 000000000000fc40 ffff881ec5263af8 Call Trace: [&lt;ffffffff8129507e&gt;] ? number+0x2ee/0x320 [&lt;ffffffff8151ecc5&gt;] rwsem_down_failed_common+0x95/0x1d0 [&lt;ffffffff8151ee23&gt;] rwsem_down_write_failed+0x23/0x30 [&lt;ffffffff812992f3&gt;] call_rwsem_down_write_failed+0x13/0x20 [&lt;ffffffff8151e322&gt;] ? down_write+0x32/0x40 [&lt;ffffffffa09d543e&gt;] dqacq_handler+0x35e/0xd20 [lquota] [&lt;ffffffffa07b8486&gt;] ? __req_capsule_get+0x176/0x750 [ptlrpc] [&lt;ffffffffa07921e0&gt;] ? lustre_swab_qdata+0x0/0x30 [ptlrpc] [&lt;ffffffffa075e1d8&gt;] target_handle_dqacq_callback+0x668/0xb90 [ptlrpc] [&lt;ffffffffa09d50e0&gt;] ? dqacq_handler+0x0/0xd20 [lquota] [&lt;ffffffffa076df87&gt;] ldlm_callback_handler+0xa17/0x1ff0 [ptlrpc] [&lt;ffffffffa0503ea1&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [&lt;ffffffffa04ff4a4&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [&lt;ffffffffa079fb4e&gt;] ptlrpc_main+0xc4e/0x1a40 [ptlrpc] [&lt;ffffffffa079ef00&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] [&lt;ffffffff8100c0ca&gt;] child_rip+0xa/0x20 [&lt;ffffffffa079ef00&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] [&lt;ffffffffa079ef00&gt;] ? ptlrpc_main+0x0/0x1a40 [ptlrpc] [&lt;ffffffff8100c0c0&gt;] ? child_rip+0x0/0x20 INFO: task ldlm_cb_01:6303 blocked <span class="code-keyword">for</span> more than 120 seconds. <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. ldlm_cb_01 D 000000000000000d 0 6303 2 0x00000080 ffff881ec5267b20 0000000000000046 0000000000000000 000000ab00000000 ffff881ec5267b50 ffffffff8129507e ffff881ec5267ad0 000000005c2ae174 ffff881ec5263098 ffff881ec5267fd8 000000000000fc40 ffff881ec5263098 </pre> </div></div> lustre: 2.1.5<br/> kernel: 2.6.32-279.19.1.el6.20130516.x86_64.lustre215<br/> build: 2nasS_ofed154<br/> <br/> SRC at <a href="https://github.com/jlan/lustre-nas">https://github.com/jlan/lustre-nas</a><br/> LU-5052

threads stuck in jbd2_journal_start

Bug Major Resolved Duplicate Zhenyu Xu Mahmoud Hanafi Mon, 12 May 2014 20:34:06 +0000 Thu, 30 Apr 2015 23:58:55 +0000 Thu, 30 Apr 2015 23:58:55 +0000 Lustre 2.4.1 0 5 <p>Bobijam</p> <p>Does this appear to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4794" title="MDS threads all stuck in jbd2_journal_start" class="issue-link" data-issue-key="LU-4794"><del>LU-4794</del></a>?</p> <p>Peter</p> <p>Bobi, <br/> according to the full stacks traces dumped in dmesg, it looks more like a dup of <a href="https://jira.whamcloud.com/browse/LU-4271" title="mds load goes very high and filesystem hangs after mounting mdt" class="issue-link" data-issue-key="LU-4271"><del>LU-4271</del></a>, still not proven has being a dup of <a href="https://jira.whamcloud.com/browse/LU-4794" title="MDS threads all stuck in jbd2_journal_start" class="issue-link" data-issue-key="LU-4794"><del>LU-4794</del></a> itself, what do you think ?</p> <p><a href="https://jira.whamcloud.com/browse/LU-4794" title="MDS threads all stuck in jbd2_journal_start" class="issue-link" data-issue-key="LU-4794"><del>LU-4794</del></a> relates to llog handling get/journal transaction deadlock. And in <a href="https://jira.whamcloud.com/browse/LU-4794" title="MDS threads all stuck in jbd2_journal_start" class="issue-link" data-issue-key="LU-4794"><del>LU-4794</del></a> bt-all.merged.txt, there also have similar thread stack trace just like the 2 threads you posted here.</p> <p>Please close</p> <p>ok - thanks Mahmoud</p> Duplicate LU-4794 Development Rank 1|hzwmd3: Rank (Obsolete) 13955 Severity