https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1276 Lustre <p>The MDS on a classified production 2.1 lustre cluster got stuck today. The symptoms were high load (800+), but very little CPU usage.</p> <p>Almost all of the lustre threads were stuck in jbd2_journal_start, while the jbd2/sda thread is stuck in<br/> jbd2_journal_commit_transaction. There is zero I/O going to disk.</p> <p>There's one thread that stands out as a suspect, as it's not in jbd2_journal_start but seems to be handling an unlink. Perhaps it got stuck waiting on a semaphore while holding an open transaction with jbd2. Its stack trace looks like this:</p> <p>COMMAND: "mdt_152"<br/> schedule<br/> rwsem_down_failed_common<br/> rwsem_down_read_failed<br/> call_rwsem_down_read_failed<br/> llog_cat_current_log.clone.0<br/> llog_cat_add_rec<br/> llog_obd_origin_add<br/> llog_add<br/> lov_llog_origin_add<br/> llog_add<br/> mds_llog_origin_add<br/> llog_add<br/> mds_llog_add_unlink<br/> mds_log_op_unlink<br/> mdd_unlink_log<br/> mdd_object_kill<br/> mdd_finish_unlink<br/> mdd_unlink<br/> cml_unlink<br/> mdt_reint_unlink<br/> mdt_reint_rec<br/> mdt_reint_internal<br/> mdt_reint<br/> mdt_handle_common<br/> mdt_regular_handle<br/> ptlrpc_main<br/> kernel_thread</p> <a href="https://github.com/chaos/lustre/commits/2.1.1-llnl">https://github.com/chaos/lustre/commits/2.1.1-llnl</a> LU-1276

MDS threads all stuck in jbd2_journal_start

Bug Critical Resolved Cannot Reproduce Oleg Drokin Ned Bass Fri, 30 Mar 2012 20:56:36 +0000 Thu, 5 May 2016 02:56:10 +0000 Tue, 30 Apr 2013 22:52:11 +0000 Lustre 2.1.0 0 5 <p>Oleg</p> <p>Could you please look into this one?</p> <p>Thanks</p> <p>Peter</p> <p>Is this <a href="https://jira.whamcloud.com/browse/LU-81" title="Some JBD2 journaling deadlock at BULL" class="issue-link" data-issue-key="LU-81"><del>LU-81</del></a>, by any chance, or another bug?</p> <p>Yes, it does look like lu-81 from this information.</p> <p>Did you get another thread in a state like this also reported as hung?<br/> PID: 26299 TASK: ffff88047d851620 CPU: 28 COMMAND: "llog_process_th"<br/> #0 <span class="error">&#91;ffff880998a65900&#93;</span> schedule at ffffffff81452851<br/> #1 <span class="error">&#91;ffff880998a659c8&#93;</span> start_this_handle at ffffffffa08ec0d7<br/> #2 <span class="error">&#91;ffff880998a65a88&#93;</span> jbd2_journal_start at ffffffffa08ec520<br/> #3 <span class="error">&#91;ffff880998a65ad8&#93;</span> ldiskfs_journal_start_sb at ffffffffa0936fb8<br/> #4 <span class="error">&#91;ffff880998a65ae8&#93;</span> fsfilt_ldiskfs_write_record at ffffffffa098a0fc<br/> #5 <span class="error">&#91;ffff880998a65b68&#93;</span> llog_lvfs_write_blob at ffffffffa050917c<br/> #6 <span class="error">&#91;ffff880998a65c18&#93;</span> llog_lvfs_write_rec at ffffffffa050a722<br/> #7 <span class="error">&#91;ffff880998a65cf8&#93;</span> llog_cancel_rec at ffffffffa05010a4<br/> #8 <span class="error">&#91;ffff880998a65d58&#93;</span> llog_cat_cancel_records at ffffffffa0505de2<br/> #9 <span class="error">&#91;ffff880998a65de8&#93;</span> llog_changelog_cancel_cb at ffffffffa099ec12<br/> #10 <span class="error">&#91;ffff880998a65e68&#93;</span> llog_process_thread at ffffffffa0503573<br/> #11 <span class="error">&#91;ffff880998a65f48&#93;</span> kernel_thread at ffffffff8100d1aa</p> <p>I will check.</p> <p>I checked and couldn't find any process in this state.</p> <p>It looks like the only thread that is in llog_cancel_rec is ldlm_cn_41 in this backtrace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>schedule start_this_handle jbd2_journal_restart ldiskfs_truncate_restart_trans ldiskfs_ext_truncate ldiskfs_truncate ldiskfs_delete_inode generic_delete_inode generic_drop_inode iput mds_obd_destroy llog_lvfs_destroy llog_cancel_rec llog_cat_cancel_records llog_origin_handle_cancel ldlm_cancel_handler ptlrpc_main kernel_thread </pre> </div></div> <p>Yes, that does match the <a href="https://jira.whamcloud.com/browse/LU-81" title="Some JBD2 journaling deadlock at BULL" class="issue-link" data-issue-key="LU-81"><del>LU-81</del></a> deadlock description.</p> <p>Ok, I have pulled the fix from <a href="https://jira.whamcloud.com/browse/LU-81" title="Some JBD2 journaling deadlock at BULL" class="issue-link" data-issue-key="LU-81"><del>LU-81</del></a> into the 2.1.1-llnl branch.</p> <p>Believed to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-81" title="Some JBD2 journaling deadlock at BULL" class="issue-link" data-issue-key="LU-81"><del>LU-81</del></a></p> <p>We had a repeat of this bug, even though we're running the <a href="https://jira.whamcloud.com/browse/LU-81" title="Some JBD2 journaling deadlock at BULL" class="issue-link" data-issue-key="LU-81"><del>LU-81</del></a> patch. We have a crash dump I can upload if anyone wants to take a look.</p> <p>There is one thread stuck in llog_cancel_rec()</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 5892 TASK: ffff880776d34040 CPU: 7 COMMAND: "ldlm_cn_57" #0 [ffff8807b50a94e0] schedule at ffffffff814ef052 #1 [ffff8807b50a95a8] start_this_handle at ffffffffa0a9e072 [jbd2] #2 [ffff8807b50a9668] jbd2_journal_restart at ffffffffa0a9e3c1 [jbd2] #3 [ffff8807b50a96b8] ldiskfs_truncate_restart_trans at ffffffffa0ac28ba [ldiskfs] #4 [ffff8807b50a96e8] ldiskfs_clear_blocks at ffffffffa0ac7bfd [ldiskfs] #5 [ffff8807b50a9748] ldiskfs_free_data at ffffffffa0ac7de4 [ldiskfs] #6 [ffff8807b50a97a8] ldiskfs_free_branches at ffffffffa0ac8023 [ldiskfs] #7 [ffff8807b50a9808] ldiskfs_free_branches at ffffffffa0ac7f16 [ldiskfs] #8 [ffff8807b50a9868] ldiskfs_truncate at ffffffffa0ac8629 [ldiskfs] #9 [ffff8807b50a9988] ldiskfs_delete_inode at ffffffffa0ac99a0 [ldiskfs] #10 [ffff8807b50a99a8] generic_delete_inode at ffffffff81192b6e #11 [ffff8807b50a99d8] generic_drop_inode at ffffffff81192cc5 #12 [ffff8807b50a99f8] iput at ffffffff81191b12 #13 [ffff8807b50a9a18] mds_obd_destroy at ffffffffa0b90161 [mds] #14 [ffff8807b50a9b58] llog_lvfs_destroy at ffffffffa05afd62 [obdclass] #15 [ffff8807b50a9c28] llog_cancel_rec at ffffffffa05a7acf [obdclass] #16 [ffff8807b50a9c58] llog_cat_cancel_records at ffffffffa05abfc1 [obdclass] #17 [ffff8807b50a9cb8] llog_origin_handle_cancel at ffffffffa072cef6 [ptlrpc] #18 [ffff8807b50a9db8] ldlm_cancel_handler at ffffffffa06fd2cf [ptlrpc] #19 [ffff8807b50a9df8] ptlrpc_main at ffffffffa0724ad1 [ptlrpc] #20 [ffff8807b50a9f48] kernel_thread at ffffffff8100c14a </pre> </div></div> <p>One thread in llog_cat_add_rec():</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 5119 TASK: ffff880831f10080 CPU: 9 COMMAND: "mdt_78" #0 [ffff8806e6e7b568] schedule at ffffffff814ef052 #1 [ffff8806e6e7b630] rwsem_down_failed_common at ffffffff814f1605 #2 [ffff8806e6e7b690] rwsem_down_read_failed at ffffffff814f1796 #3 [ffff8806e6e7b6d0] call_rwsem_down_read_failed at ffffffff81278d04 #4 [ffff8806e6e7b738] llog_cat_current_log.clone.0 at ffffffffa05aa715 [obdclass] #5 [ffff8806e6e7b7d8] llog_cat_add_rec at ffffffffa05ac18a [obdclass] #6 [ffff8806e6e7b828] llog_obd_origin_add at ffffffffa05b09a6 [obdclass] #7 [ffff8806e6e7b858] llog_add at ffffffffa05b0b01 [obdclass] #8 [ffff8806e6e7b898] lov_llog_origin_add at ffffffffa091d0c4 [lov] #9 [ffff8806e6e7b918] llog_add at ffffffffa05b0b01 [obdclass] #10 [ffff8806e6e7b958] mds_llog_origin_add at ffffffffa0b915d3 [mds] #11 [ffff8806e6e7b9a8] llog_add at ffffffffa05b0b01 [obdclass] #12 [ffff8806e6e7b9e8] mds_llog_add_unlink at ffffffffa0b91a95 [mds] #13 [ffff8806e6e7ba38] mds_log_op_unlink at ffffffffa0b91f84 [mds] #14 [ffff8806e6e7ba98] mdd_unlink_log at ffffffffa0bc0e4e [mdd] #15 [ffff8806e6e7bac8] mdd_object_kill at ffffffffa0bb8242 [mdd] #16 [ffff8806e6e7bb08] mdd_finish_unlink at ffffffffa0bcb5f6 [mdd] #17 [ffff8806e6e7bb48] mdd_unlink at ffffffffa0bd010a [mdd] #18 [ffff8806e6e7bc08] cml_unlink at ffffffffa0ce4e78 [cmm] #19 [ffff8806e6e7bc58] mdt_reint_unlink at ffffffffa0c3aded [mdt] #20 [ffff8806e6e7bce8] mdt_reint_rec at ffffffffa0c392a0 [mdt] #21 [ffff8806e6e7bd18] mdt_reint_internal at ffffffffa0c34098 [mdt] #22 [ffff8806e6e7bd68] mdt_reint at ffffffffa0c34354 [mdt] #23 [ffff8806e6e7bd98] mdt_handle_common at ffffffffa0c287ad [mdt] #24 [ffff8806e6e7bde8] mdt_regular_handle at ffffffffa0c29405 [mdt] #25 [ffff8806e6e7bdf8] ptlrpc_main at ffffffffa0724ad1 [ptlrpc] #26 [ffff8806e6e7bf48] kernel_thread at ffffffff8100c14a </pre> </div></div> <p>And 115 threads stuck in jbd2_journal_start(), i.e.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 4075 TASK: ffff880231f93540 CPU: 15 COMMAND: "mdt_01" #0 [ffff8801018cf6a0] schedule at ffffffff814ef052 #1 [ffff8801018cf768] start_this_handle at ffffffffa0a9e072 [jbd2] #2 [ffff8801018cf828] jbd2_journal_start at ffffffffa0a9e4f0 [jbd2] #3 [ffff8801018cf878] ldiskfs_journal_start_sb at ffffffffa0aec098 [ldiskfs] #4 [ffff8801018cf888] osd_trans_start at ffffffffa0cb5e0c [osd_ldiskfs] #5 [ffff8801018cf8c8] mdd_trans_start at ffffffffa0bd9753 [mdd] #6 [ffff8801018cf8e8] mdd_create at ffffffffa0bd086b [mdd] #7 [ffff8801018cfa28] cml_create at ffffffffa0ce54d8 [cmm] #8 [ffff8801018cfa78] mdt_reint_open at ffffffffa0c4c924 [mdt] #9 [ffff8801018cfb68] mdt_reint_rec at ffffffffa0c392a0 [mdt] #10 [ffff8801018cfb98] mdt_reint_internal at ffffffffa0c34098 [mdt] #11 [ffff8801018cfbe8] mdt_intent_reint at ffffffffa0c34555 [mdt] #12 [ffff8801018cfc38] mdt_intent_policy at ffffffffa0c2fe29 [mdt] #13 [ffff8801018cfc88] ldlm_lock_enqueue at ffffffffa06dfb42 [ptlrpc] #14 [ffff8801018cfcf8] ldlm_handle_enqueue0 at ffffffffa06fe906 [ptlrpc] #15 [ffff8801018cfd68] mdt_enqueue at ffffffffa0c2fa9a [mdt] #16 [ffff8801018cfd98] mdt_handle_common at ffffffffa0c287ad [mdt] #17 [ffff8801018cfde8] mdt_regular_handle at ffffffffa0c29405 [mdt] #18 [ffff8801018cfdf8] ptlrpc_main at ffffffffa0724ad1 [ptlrpc] #19 [ffff8801018cff48] kernel_thread at ffffffff8100c14a </pre> </div></div> <p>Reopening ticket - not a duplicate of LU81</p> <p>I guess the other candidate for this issue is <a href="https://jira.whamcloud.com/browse/LU-1648" title="MDS Crash" class="issue-link" data-issue-key="LU-1648"><del>LU-1648</del></a>, can you add a patch from it as well please?</p> <p>It sounds like the patch from <a href="https://jira.whamcloud.com/browse/LU-1648" title="MDS Crash" class="issue-link" data-issue-key="LU-1648"><del>LU-1648</del></a> is needed on b2_1.</p> <p>It looks like the <a href="https://jira.whamcloud.com/browse/LU-1648" title="MDS Crash" class="issue-link" data-issue-key="LU-1648"><del>LU-1648</del></a> fix landed on before 2.1.4 in <a href="http://review.whamcloud.com/4743" class="external-link" target="_blank" rel="nofollow noopener">change 4743</a>. I think we can close this until we see it again.</p> <p>ok thanks Chris</p> <p>One of the Bull customers (TGCC) had the same deadlock twice during the past six months: one thread is stuck in jbd2_journal_commit_transaction() and many other thread are stuck in jbd2_journal_start().</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 29225 TASK: ffff88107c3bb040 CPU: 15 COMMAND: "jbd2/dm-2-8" #0 [ffff88107a343c60] schedule at ffffffff81485765 0000001 [ffff88107a343d28] jbd2_journal_commit_transaction at ffffffffa006a94f [jbd2] 0000002 [ffff88107a343e68] kjournald2 at ffffffffa0070c08 [jbd2] 0000003 [ffff88107a343ee8] kthread at ffffffff8107b5f6 0000004 [ffff88107a343f48] kernel_thread at ffffffff8100412a </pre> </div></div> <p>and most of the threads:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 15585 TASK: ffff88063062a790 CPU: 0 COMMAND: "mdt_503" PID: 15586 TASK: ffff88063062a040 CPU: 23 COMMAND: "mdt_504" PID: 15587 TASK: ffff88020f3ad7d0 CPU: 30 COMMAND: "mdt_505" PID: 29286 TASK: ffff88087505e790 CPU: 25 COMMAND: "mdt_01" ... #0 [ffff881949c078f0] schedule at ffffffff81485765 0000001 [ffff881949c079b8] start_this_handle at ffffffffa006908a [jbd2] 0000002 [ffff881949c07a78] jbd2_journal_start at ffffffffa0069500 [jbd2] 0000003 [ffff881949c07ac8] ldiskfs_journal_start_sb at ffffffffa0451ca8 [ldiskfs] 0000004 [ffff881949c07ad8] osd_trans_start at ffffffffa0d4a324 [osd_ldiskfs] 0000005 [ffff881949c07b18] mdd_trans_start at ffffffffa0c4c4e3 [mdd] 0000006 [ffff881949c07b38] mdd_unlink at ffffffffa0c401eb [mdd] 0000007 [ffff881949c07bf8] cml_unlink at ffffffffa0d82e07 [cmm] 0000008 [ffff881949c07c38] mdt_reint_unlink at ffffffffa0cba0f4 [mdt] 0000009 [ffff881949c07cb8] mdt_reint_rec at ffffffffa0cb7cb1 [mdt] 0000010 [ffff881949c07cd8] mdt_reint_internal at ffffffffa0caeed4 [mdt] 0000011 [ffff881949c07d28] mdt_reint at ffffffffa0caf2b4 [mdt] 0000012 [ffff881949c07d48] mdt_handle_common at ffffffffa0ca3762 [mdt] 0000013 [ffff881949c07d98] mdt_regular_handle at ffffffffa0ca4655 [mdt] 0000014 [ffff881949c07da8] ptlrpc_main at ffffffffa071f4f6 [ptlrpc] 0000015 [ffff881949c07f48] kernel_thread at ffffffff8100412a </pre> </div></div> <p>The are running lustre 2.1.6 which contains <a href="http://review.whamcloud.com/4743" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/4743</a> from <a href="https://jira.whamcloud.com/browse/LU-1648" title="MDS Crash" class="issue-link" data-issue-key="LU-1648"><del>LU-1648</del></a>.</p> <p>I attach two files containing the dmseg and the crash back trace of all threads.<br/> could you reopen this ticket, as it was closed with "Cannot Reproduce". </p> Duplicate Development Rank 1|hzvpt3: Rank (Obsolete) 8047 Severity