https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15800 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 74368 TASK: ffff9600eaeac740 CPU: 9 COMMAND: "ll_ost_io02_069" #0 [ffffa3f1a7a57830] __schedule at ffffffff9034e1d4 #1 [ffffa3f1a7a578c8] schedule at ffffffff9034e648 #2 [ffffa3f1a7a578d8] rwsem_down_read_slowpath at ffffffff903511d0 #3 [ffffa3f1a7a57978] osd_read_lock at ffffffffc1a3379d [osd_ldiskfs] &lt;-- rc = dt_trans_start_local(env, ofd-&gt;ofd_osd , th); ofd_read_lock(env, ofd_obj); #4 [ffffa3f1a7a57998] ofd_write_attr_set at ffffffffc186b6cc [ofd] #5 [ffffa3f1a7a57a00] ofd_commitrw_write at ffffffffc186c812 [ofd] #6 [ffffa3f1a7a57aa0] ofd_commitrw at ffffffffc18721f1 [ofd] #7 [ffffa3f1a7a57b60] finish_wait at ffffffff8fb2e5ac #8 [ffffa3f1a7a57bd8] tgt_brw_write at ffffffffc1255544 [ptlrpc] </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 73559 TASK: ffff9601653a97c0 CPU: 11 COMMAND: "ll_ost02_046" #0 [ffffa3f1a0817970] __schedule at ffffffff9034e1d4 #1 [ffffa3f1a0817a08] schedule at ffffffff9034e648 #2 [ffffa3f1a0817a18] wait_transaction_locked at ffffffffc0ad2089 [jbd2] #3 [ffffa3f1a0817a68] add_transaction_credits at ffffffffc0ad21c4 [jbd2] #4 [ffffa3f1a0817ac0] start_this_handle at ffffffffc0ad250a [jbd2] #5 [ffffa3f1a0817b40] jbd2__journal_restart at ffffffffc0ad2ad0 [jbd2] #6 [ffffa3f1a0817b80] osd_fallocate_preallocate at ffffffffc1a5b6d2 [osd_ldiskfs] #7 [ffffa3f1a0817c18] osd_fallocate at ffffffffc1a5b98d [osd_ldiskfs] &lt;-- ofd_trans_start(env, ofd, fo, th); ofd_write_lock(env, fo); #8 [ffffa3f1a0817c50] ofd_object_fallocate at ffffffffc18682f9 [ofd] #9 [ffffa3f1a0817cb8] ofd_fallocate_hdl at ffffffffc185912f [ofd] #10 [ffffa3f1a0817d50] tgt_request_handle at ffffffffc1256a53 [ptlrpc]</pre> </div></div> <p>The deadlock was added by :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> Commit: 93f700ca241a98630fc5ff19a041e35fbdbf0385 Author: Arshad Hussain &lt;arshad.super@gmail.com&gt; Committer: Oleg Drokin &lt;green@whamcloud.com&gt; Author Date: Thu 10 Sep 2020 02:18:13 AM EEST Committer Date: Thu 29 Oct 2020 06:28:42 AM EET LU-13765 osd-ldiskfs: Extend credit correctly for fallocate </pre> </div></div> LU-15800

Fallocate causes transaction deadlock

Bug Critical Resolved Fixed Arshad Hussain Andriy Skulysh ost Thu, 28 Apr 2022 09:24:10 +0000 Wed, 2 Aug 2023 13:08:24 +0000 Mon, 6 Jun 2022 13:27:21 +0000 Lustre 2.16.0 Lustre 2.15.4 0 7 <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=jhammond" class="user-hover" rel="jhammond">jhammond</a>, it isn't a duplicate of <a href="https://jira.whamcloud.com/browse/LU-14214" title="incorrect locking in ofd_object_fallocate()" class="issue-link" data-issue-key="LU-14214"><del>LU-14214</del></a>, the locking is correct in the code but it is violated  by jbd2__journal_restart(). It shouldn't be called under ofd_write_lock()</p> <p>Arshad</p> <p>Is this something that you are able to look into?</p> <p>Peter</p> <p>Hi Peter, I am looking into it.</p> <p>Thanks</p> <p>Andriy,</p> <p>Is there a test case(or manual steps) that can trigger this issue? Environment details would also help. (how large was the fallocate?) At least in my case, It would greatly help to have such details/reproducer. I tried to reproduce the bug running standard sanity/sanityn test-case over loop but failed to reproduce the deadlock. At-least the standard test-case does not catch/trigger this. With the stack trace you provided, it looks like there are two threads one doing fallocate(standard prealloc) other doing a write(eg dd). ?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>&gt;...in the code but it is violated  by jbd2__journal_restart(). It shouldn't be called under ofd_write_lock()</pre> </div></div> <p>Sorry, I thought this is fine under ofd lock. Can you please explain in more details? </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>osd_extend_restart_trans()     -&gt;ldiskfs_journal_restart()         -&gt; jbd2_journal_restart()             -&gt; jbd2__journal_restart()</pre> </div></div> <p> </p> <p>Thanks</p> <p>Given that this issue existed in 2.14, I think that it should be ok to descope it from 2.15.0 and include in a future 2.15.x maintenance release.</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47268" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47268</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15800" title="Fallocate causes transaction deadlock" class="issue-link" data-issue-key="LU-15800"><del>LU-15800</del></a> ofd: take a read lock for fallocate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ce637a38fd863f07c1e9a35f9a7c0731d858c23e</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47268/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47268/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15800" title="Fallocate causes transaction deadlock" class="issue-link" data-issue-key="LU-15800"><del>LU-15800</del></a> ofd: take a read lock for fallocate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5fae80066162ea637c8649f6439fc14e1d9a7cf8</p> <p>Landed for 2.16</p> <p>"Xing Huang &lt;hxing@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51702" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51702</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15800" title="Fallocate causes transaction deadlock" class="issue-link" data-issue-key="LU-15800"><del>LU-15800</del></a> ofd: take a read lock for fallocate<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: a29082fa9985ce97d3e02b8c1009161e54f11f9a</p> <p>We also hit this OSS deadlock with Lustre 2.15.3 yesterday. The backtraces seem to match:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PID: 61557 TASK: ffff919ef006a100 CPU: 39 COMMAND: "ll_ost_io00_078" #0 [ffff919cea263778] __schedule at ffffffff92db78d8 #1 [ffff919cea2637e0] schedule at ffffffff92db7ca9 #2 [ffff919cea2637f0] rwsem_down_read_failed at ffffffff92db9705 #3 [ffff919cea263878] call_rwsem_down_read_failed at ffffffff929ae568 #4 [ffff919cea2638c8] down_read at ffffffff92db7120 #5 [ffff919cea2638e0] osd_read_lock at ffffffffc16d4e7c [osd_ldiskfs] #6 [ffff919cea263908] ofd_write_attr_set at ffffffffc1863129 [ofd] #7 [ffff919cea263978] ofd_commitrw_write at ffffffffc1863fd2 [ofd] #8 [ffff919cea263a30] ofd_commitrw at ffffffffc18698e0 [ofd] #9 [ffff919cea263ac0] tgt_brw_write at ffffffffc140c695 [ptlrpc] #10 [ffff919cea263ca8] tgt_request_handle at ffffffffc140f25f [ptlrpc] #11 [ffff919cea263d38] ptlrpc_server_handle_request at ffffffffc13b8aa3 [ptlrpc] #12 [ffff919cea263df0] ptlrpc_main at ffffffffc13ba734 [ptlrpc] #13 [ffff919cea263ec8] kthread at ffffffff926cb621 #14 [ffff919cea263f50] ret_from_fork_nospec_begin at ffffffff92dc51dd PID: 40363 TASK: ffff915f2d945280 CPU: 10 COMMAND: "ll_ost00_123" #0 [ffff9159d062f8f0] __schedule at ffffffff92db78d8 #1 [ffff9159d062f958] schedule at ffffffff92db7ca9 #2 [ffff9159d062f968] wait_transaction_locked at ffffffffc03ca085 [jbd2] #3 [ffff9159d062f9c0] add_transaction_credits at ffffffffc03ca378 [jbd2] #4 [ffff9159d062fa20] start_this_handle at ffffffffc03ca601 [jbd2] #5 [ffff9159d062fab8] jbd2__journal_restart at ffffffffc03cacf2 [jbd2] #6 [ffff9159d062faf8] jbd2_journal_restart at ffffffffc03cad63 [jbd2] #7 [ffff9159d062fb08] osd_extend_restart_trans at ffffffffc1700d8c [osd_ldiskfs] #8 [ffff9159d062fb28] osd_fallocate at ffffffffc1702dc4 [osd_ldiskfs] #9 [ffff9159d062fbb0] ofd_object_fallocate at ffffffffc185fb4f [ofd] #10 [ffff9159d062fc18] ofd_fallocate_hdl at ffffffffc1848835 [ofd] #11 [ffff9159d062fca8] tgt_request_handle at ffffffffc140f25f [ptlrpc] #12 [ffff9159d062fd38] ptlrpc_server_handle_request at ffffffffc13b8aa3 [ptlrpc] #13 [ffff9159d062fdf0] ptlrpc_main at ffffffffc13ba734 [ptlrpc] #14 [ffff9159d062fec8] kthread at ffffffff926cb621 #15 [ffff9159d062ff50] ret_from_fork_nospec_begin at ffffffff92dc51dd </pre> </div></div> <p>We will try the proposed patch (thanks!).</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51702/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51702/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15800" title="Fallocate causes transaction deadlock" class="issue-link" data-issue-key="LU-15800"><del>LU-15800</del></a> ofd: take a read lock for fallocate<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: 8299b3fd77ebcc372b5d929eaa08231fc703c431</p> Duplicate LU-16966 Related LU-13765 LU-14214 Development Rank 1|i02ogn: Rank (Obsolete) 9223372036854775807 Severity