https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13072 Lustre <p>We found an OSS on Fir (2.12.3_4) heavily loaded and with threads hung this morning. This started during last night. We took a crash dump that I uploaded to the FTP as <tt>vmcore_fir-io8-s1_2019-12-12-07-42-58</tt> along with kernel-debuginfo (<tt>kernel-debuginfo-3.10.0-957.27.2.el7_lustre.pl2.x86_64.rpm</tt> and <tt>kernel-debuginfo-common-x86_64-3.10.0-957.27.2.el7_lustre.pl2.x86_64.rpm</tt>).</p> <p>The last messages on the console before the crash dump were:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[264018.187097] LNet: Service thread pid 33911 was inactive for 1200.94s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [264018.204204] LNet: Skipped 1 previous similar message [264018.209269] Pid: 33911, comm: ll_ost03_106 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [264018.219804] Call Trace: [264018.222354] [&lt;ffffffffc155deb3&gt;] ofd_create_hdl+0xcb3/0x20e0 [ofd] [264018.228754] [&lt;ffffffffc0f5036a&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] [264018.235831] [&lt;ffffffffc0ef724b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [264018.243637] [&lt;ffffffffc0efabac&gt;] ptlrpc_main+0xb2c/0x1460 [ptlrpc] [264018.250067] [&lt;ffffffffa7ac2e81&gt;] kthread+0xd1/0xe0 [264018.255066] [&lt;ffffffffa8177c24&gt;] ret_from_fork_nospec_begin+0xe/0x21 [264018.261642] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [264018.266754] LustreError: dumping log to /tmp/lustre-log.1576165038.33911 [264126.221225] LNet: Service thread pid 34617 was inactive for 200.36s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [264126.238268] Pid: 34617, comm: ll_ost03_111 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [264126.248785] Call Trace: [264126.251360] [&lt;ffffffffc02c3085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] [264126.258376] [&lt;ffffffffc02c3368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] [264126.265544] [&lt;ffffffffc02c35e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] [264126.272211] [&lt;ffffffffc02c3a93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] [264126.278951] [&lt;ffffffffc131b309&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [264126.286491] [&lt;ffffffffc1312ad3&gt;] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs] [264126.293573] [&lt;ffffffffa7cb026a&gt;] dqget+0x3fa/0x450 [264126.298588] [&lt;ffffffffa7cb1074&gt;] dquot_get_dqblk+0x14/0x1f0 [264126.304371] [&lt;ffffffffc143c6d5&gt;] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs] [264126.311997] [&lt;ffffffffc13a380d&gt;] lquotactl_slv+0x27d/0x9d0 [lquota] [264126.318488] [&lt;ffffffffc1556a3c&gt;] ofd_quotactl+0x13c/0x380 [ofd] [264126.324630] [&lt;ffffffffc0f5036a&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] [264126.331681] [&lt;ffffffffc0ef724b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [264126.339496] [&lt;ffffffffc0efabac&gt;] ptlrpc_main+0xb2c/0x1460 [ptlrpc] [264126.345912] [&lt;ffffffffa7ac2e81&gt;] kthread+0xd1/0xe0 [264126.350926] [&lt;ffffffffa8177c24&gt;] ret_from_fork_nospec_begin+0xe/0x21 [264126.357490] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [264126.362604] LustreError: dumping log to /tmp/lustre-log.1576165146.34617 [264136.461417] Pid: 34429, comm: ll_ost03_110 3.10.0-957.27.2.el7_lustre.pl2.x86_64 #1 SMP Thu Nov 7 15:26:16 PST 2019 [264136.471940] Call Trace: [264136.474507] [&lt;ffffffffc02c3085&gt;] wait_transaction_locked+0x85/0xd0 [jbd2] [264136.481514] [&lt;ffffffffc02c3368&gt;] add_transaction_credits+0x268/0x2f0 [jbd2] [264136.488699] [&lt;ffffffffc02c35e1&gt;] start_this_handle+0x1a1/0x430 [jbd2] [264136.495351] [&lt;ffffffffc02c3a93&gt;] jbd2__journal_start+0xf3/0x1f0 [jbd2] [264136.502103] [&lt;ffffffffc131b309&gt;] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs] [264136.509629] [&lt;ffffffffc1312ad3&gt;] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs] [264136.516725] [&lt;ffffffffa7cb026a&gt;] dqget+0x3fa/0x450 [264136.521728] [&lt;ffffffffa7cb1074&gt;] dquot_get_dqblk+0x14/0x1f0 [264136.527514] [&lt;ffffffffc143c6d5&gt;] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs] [264136.535127] [&lt;ffffffffc13a380d&gt;] lquotactl_slv+0x27d/0x9d0 [lquota] [264136.541618] [&lt;ffffffffc1556a3c&gt;] ofd_quotactl+0x13c/0x380 [ofd] [264136.547746] [&lt;ffffffffc0f5036a&gt;] tgt_request_handle+0xaea/0x1580 [ptlrpc] [264136.554806] [&lt;ffffffffc0ef724b&gt;] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [264136.562610] [&lt;ffffffffc0efabac&gt;] ptlrpc_main+0xb2c/0x1460 [ptlrpc] [264136.569039] [&lt;ffffffffa7ac2e81&gt;] kthread+0xd1/0xe0 [264136.574042] [&lt;ffffffffa8177c24&gt;] ret_from_fork_nospec_begin+0xe/0x21 [264136.580632] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [264136.585737] LustreError: dumping log to /tmp/lustre-log.1576165156.34429 [264349.590511] SysRq : Trigger a crash </pre> </div></div> <p>Attaching:</p> <ul> <li>vmcore-dmesg-txt as <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/34018/34018_vmcore-dmesg_fir-io8-s1_2019-12-12-07-42-58.txt" title="vmcore-dmesg_fir-io8-s1_2019-12-12-07-42-58.txt attached to LU-13072">vmcore-dmesg_fir-io8-s1_2019-12-12-07-42-58.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></li> <li>output of "foreach bt" as <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/34017/34017_foreach_bt_fir-io8-s1_2019-12-12-07-42-58.txt" title="foreach_bt_fir-io8-s1_2019-12-12-07-42-58.txt attached to LU-13072">foreach_bt_fir-io8-s1_2019-12-12-07-42-58.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></li> </ul> <p>We restarted the OSS and a few OSTs couldn't complete their recovery (stuck at 0s) but a manual <tt>abort_recovery</tt> did work.</p> <p>Thanks,<br/> Stephane</p> lustre-2.12.3_4_g142b4d4-1.el7.x86_64 3.10.0-957.27.2.el7_lustre.pl2.x86_64 MOFED 4.7 LU-13072

High OSS load due to possible deadlock w/ ofd_create_hdl and ofd_quotactl backtraces

Bug Major Resolved Fixed Yang Sheng Stephane Thiell Thu, 12 Dec 2019 19:11:29 +0000 Tue, 26 Jul 2022 04:01:34 +0000 Tue, 26 Jul 2022 04:01:34 +0000 Lustre 2.12.3 0 6 <p>Yang Sheng</p> <p>Could you please advise</p> <p>Thanks</p> <p>Peter</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 66168 TASK: ffff8912f5c68000 CPU: 0 COMMAND: "jbd2/md2-8" #0 [ffff8912f34cfc08] __schedule at ffffffffa816aa72 #1 [ffff8912f34cfc98] schedule at ffffffffa816af19 #2 [ffff8912f34cfca8] jbd2_journal_commit_transaction at ffffffffc02c62bc [jbd2] #3 [ffff8912f34cfe48] kjournald2 at ffffffffc02cce89 [jbd2] #4 [ffff8912f34cfec8] kthread at ffffffffa7ac2e81 #5 [ffff8912f34cff50] ret_from_fork_nospec_begin at ffffffffa8177c24 </pre> </div></div> <p>So jbd is commiting a transaction and it takes a long time so nobody else can start a new transaction for a long time. I also see many other threads waiting on IO including debugfs that I presume reads from the same device? but also many ost io threads. If there's a lot of IO contention it's somewhat understandable that IO is slow including slow transaction commits that impacts things like starting new transactions.</p> <p>Hi, Stephane,</p> <p>After investigating, I think this issue should relate to some storage issue. The all of ofd_create_hdl threads were block by wait_transaction_locked. The ofd_quotactl also same blocked on wait_transaction_locked. They are waiting to start a new trans. The transaction was block by </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 67875 TASK: ffff891241b65140 CPU: 9 COMMAND: "ll_ost_io01_019" #0 [ffff891241e2f8e8] __schedule at ffffffffa816aa72 #1 [ffff891241e2f978] schedule at ffffffffa816af19 #2 [ffff891241e2f988] osd_trans_stop at ffffffffc140fe75 [osd_ldiskfs] #3 [ffff891241e2fa40] ofd_trans_stop at ffffffffc1569c75 [ofd] #4 [ffff891241e2fa50] ofd_commitrw_write at ffffffffc1570c34 [ofd] #5 [ffff891241e2fad8] ofd_commitrw at ffffffffc157510c [ofd] #6 [ffff891241e2fb58] tgt_brw_write at ffffffffc0f5460b [ptlrpc] #7 [ffff891241e2fcd0] tgt_request_handle at ffffffffc0f5036a [ptlrpc] #8 [ffff891241e2fd58] ptlrpc_server_handle_request at ffffffffc0ef724b [ptlrpc] #9 [ffff891241e2fdf8] ptlrpc_main at ffffffffc0efabac [ptlrpc] #10 [ffff891241e2fec8] kthread at ffffffffa7ac2e81 #11 [ffff891241e2ff50] ret_from_fork_nospec_begin at ffffffffa8177c24 </pre> </div></div> <p>The thread was waiting some data to write out. It is waiting for a few threads like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 67874 TASK: ffff891241b64100 CPU: 42 COMMAND: "ll_ost_io02_019" #0 [ffff891241e2b4a8] __schedule at ffffffffa816aa72 #1 [ffff891241e2b538] schedule at ffffffffa816af19 #2 [ffff891241e2b548] bitmap_startwrite at ffffffffa7f9cc65 #3 [ffff891241e2b5c8] add_stripe_bio at ffffffffc1383471 [raid456] #4 [ffff891241e2b628] raid5_make_request at ffffffffc1388db4 [raid456] #5 [ffff891241e2b710] md_handle_request at ffffffffa7f8fa30 #6 [ffff891241e2b780] md_make_request at ffffffffa7f8fb99 #7 [ffff891241e2b7a8] generic_make_request at ffffffffa7d46e37 #8 [ffff891241e2b800] submit_bio at ffffffffa7d470e0 #9 [ffff891241e2b858] osd_submit_bio at ffffffffc14255ac [osd_ldiskfs] #10 [ffff891241e2b868] osd_do_bio at ffffffffc1427925 [osd_ldiskfs] #11 [ffff891241e2b9d0] osd_write_commit at ffffffffc142820c [osd_ldiskfs] #12 [ffff891241e2ba50] ofd_commitrw_write at ffffffffc157121e [ofd] #13 [ffff891241e2bad8] ofd_commitrw at ffffffffc157510c [ofd] #14 [ffff891241e2bb58] tgt_brw_write at ffffffffc0f5460b [ptlrpc] #15 [ffff891241e2bcd0] tgt_request_handle at ffffffffc0f5036a [ptlrpc] #16 [ffff891241e2bd58] ptlrpc_server_handle_request at ffffffffc0ef724b [ptlrpc] #17 [ffff891241e2bdf8] ptlrpc_main at ffffffffc0efabac [ptlrpc] #18 [ffff891241e2bec8] kthread at ffffffffa7ac2e81 #19 [ffff891241e2bf50] ret_from_fork_nospec_begin at ffffffffa8177c24 </pre> </div></div> <p>They were running on /dev/md2 and looks like this is a software raid6 device. I suggest you should investigate write performance for this device to measure whether it is slowly than other.</p> <p>Thanks,<br/> YangSheng</p> <p>Hi YangSheng,</p> <p>Thank you very much for your time on this, I really appreciate it.  After the reboot, all OSTs came back online and everything was fine on all targets, even on /dev/md2. I wonder if this could be some kind of mdraid deadlock of some sort, or perhaps a very transient slowdown of a raid6 array.</p> <p>Stephane</p> <p>Hi, Stephane,</p> <p>I have investigated all of thread carefully. I cannot find any clue for deadlock. I suggest you can collect disk io info(like iostat) while this issue be hit again. Then compare it with normal data(collect without this issue) to determine whether it is a storage problem. </p> <p>Thanks,<br/> YangSheng</p> <p>Hi YangSheng,</p> <p>Thanks, next time I'll try to collect disk I/O info, that's a great idea indeed. But so far, this OST has been running fine since this event.</p> <p>Best,</p> <p>Stephane</p> <p>Please reopen it while hit again.</p> Development Rank 1|i00qxz: Rank (Obsolete) 9223372036854775807 Severity