https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17125 Lustre <p>The following commit deadlock has been observed on a customer side:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 7 16:41:55 lustren002 kernel: jbd2/md65-8 D 0 259595 2 0x80004000 Sep 7 16:41:55 lustren002 kernel: Call Trace: Sep 7 16:41:55 lustren002 kernel: __schedule+0x2c4/0x700 Sep 7 16:41:55 lustren002 kernel: ? __wake_up_common_lock+0x89/0xc0 Sep 7 16:41:55 lustren002 kernel: schedule+0x38/0xa0 Sep 7 16:41:55 lustren002 kernel: jbd2_log_wait_commit+0xac/0x120 [jbd2] Sep 7 16:41:55 lustren002 kernel: ? finish_wait+0x80/0x80 Sep 7 16:41:55 lustren002 kernel: ldiskfs_sync_fs+0x1c1/0x1d0 [ldiskfs] Sep 7 16:41:55 lustren002 kernel: osd_sync+0xdf/0x180 [osd_ldiskfs] Sep 7 16:41:55 lustren002 kernel: qsd_acquire+0x468/0xdd0 [lquota] Sep 7 16:41:55 lustren002 kernel: qsd_op_begin0+0x161/0x890 [lquota] Sep 7 16:41:55 lustren002 kernel: ? __kmalloc+0x102/0x240 Sep 7 16:41:55 lustren002 kernel: qsd_reserve_or_free_quota+0x2e1/0x7b0 [lquota] Sep 7 16:41:55 lustren002 kernel: osd_reserve_or_free_quota+0xe6/0x1c0 [osd_ldiskfs] Sep 7 16:41:55 lustren002 kernel: ? lu_context_init+0xa5/0x1a0 [obdclass] Sep 7 16:41:55 lustren002 kernel: tgt_cb_last_committed+0xc2/0x5c0 [ptlrpc] Sep 7 16:41:55 lustren002 kernel: osd_trans_commit_cb+0xe6/0x300 [osd_ldiskfs] Sep 7 16:41:55 lustren002 kernel: ldiskfs_journal_commit_callback+0xa5/0xe0 [ldiskfs] Sep 7 16:41:55 lustren002 kernel: jbd2_journal_commit_transaction+0x17ba/0x19e0 [jbd2] Sep 7 16:41:55 lustren002 kernel: kjournald2+0xbd/0x270 [jbd2] </pre> </div></div> <p>Revert of "<a href="https://jira.whamcloud.com/browse/LU-14370" title=" sanity-quota test_12b: @@@@@@ FAIL: create failed, but expect success " class="issue-link" data-issue-key="LU-14370"><del>LU-14370</del></a> quota: use dt_sync() to flush pending writes" could be a potential fix for the problem.</p> LU-17125

jbd2 commit deadlock

Bug Critical Closed Duplicate WC Triage Vladimir Saveliev Mon, 18 Sep 2023 11:07:42 +0000 Wed, 25 Oct 2023 15:50:55 +0000 Wed, 25 Oct 2023 15:50:55 +0000 1 5 <p>I rather think that calling qsd_acquire() from commit callback context is a bad idea.</p> <blockquote> <p>I rather think that calling qsd_acquire() from commit callback context is a bad idea.</p></blockquote> <p>ok, then would it make sense to ask Hongchao Zhang to take a look, as that was done in:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>From: Hongchao Zhang &lt;hongchao@whamcloud.com&gt; Date: Fri, 2 Apr 2021 14:53:59 +0800 Subject: [PATCH] LU-11303 quota: enforce block quota for chgrp </pre> </div></div> <p>The issue was hit with the lustre code w/o <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> landed, that code contains an incorrect integer values conversion when tgt_cb_last_committed() calls dt_reserve_or_free_quota():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> dt_reserve_or_free_quota(&amp;temp_env, th-&gt;th_dev, th-&gt;th_reserved_quota.qrr_type, th-&gt;th_reserved_quota.qrr_id.qid_uid, th-&gt;th_reserved_quota.qrr_id.qid_gid, -th-&gt;th_reserved_quota.qrr_count, <span class="code-keyword">false</span>); </pre> </div></div> <p>the "count" parameter in dt_reserve_or_free_quota prototype is inly 4-bytes "int", while it is used to pass an 64bit value down the call stack. For enough big numbers of quota space to be released, the sign bit of the value can be lost.. the underlaying functions use sign of the passing value to determine whether it is quota "reservation" or "freeing". Eventually, in qsd_reserve_or_free_quota(), it causes wrong function to be called, qsd_op_begin0 instead of qsd_op_end0 at the transaction commit time.</p> <p>here the values from a crash dump:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>crash&gt; osd_thandle.ot_super.th_reserved_quota -x ffff95df1fc4ee00 ot_super.th_reserved_quota = { qrr_type = GRPQUOTA, qrr_id = { qid_fid = { f_seq = 0x760a83c4, f_oid = 0x0, f_ver = 0x0 }, qid_uid = 0x760a83c4, qid_gid = 0x760a83c4, qid_projid = 0x760a83c4 }, qrr_count = 0x80001000 }, crash&gt; </pre> </div></div> <p>And, <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> fixes that by getting rid of "int count" parameter in dt_reserve_or_free_quota(). So the issue should not be reproducible with the changes from <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a> patches in place.</p> <p>Closing as a dup of <a href="https://jira.whamcloud.com/browse/LU-15880" title="ASSERTION( lqe-&gt;u.se.lse_pending_write == 0 )" class="issue-link" data-issue-key="LU-15880"><del>LU-15880</del></a>.</p> Duplicate LU-15880 Development Rank 1|i03vz3: Rank (Obsolete) 9223372036854775807 Severity