https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17332 Lustre <p>This issue was created by maloo for Andreas Dilger &lt;adilger@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/58a6b07c-fb1f-4a2d-ac3c-d7578d6b134f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/58a6b07c-fb1f-4a2d-ac3c-d7578d6b134f</a></p> <p>test_820 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>trevis-28vm3 crashed during sanity test_820 [26282.338565] Lustre: server umount lustre-OST0004 complete [26282.411017] ------------[ cut here ]------------ [26282.412061] kernel BUG at fs/jbd2/transaction.c:378! [26282.413171] invalid opcode: 0000 [#1] SMP PTI [26282.414068] CPU: 1 PID: 784404 Comm: kworker/1:5 4.18.0-477.15.1.el8_lustre.x86_64 #1 [26282.416473] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [26282.435639] Call Trace: [26282.438083] jbd2__journal_start+0xee/0x1f0 [jbd2] [26282.439047] jbd2_journal_start+0x19/0x20 [jbd2] [26282.439979] flush_stashed_stats_work+0x36/0x90 [ldiskfs] [26282.441086] process_one_work+0x1a7/0x360 [26282.442753] worker_thread+0x30/0x390 [26282.444311] kthread+0x134/0x150 </pre> </div></div> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-master/4445" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master/4445</a> - 4.18.0-477.15.1.el8_8.x86_64<br/> servers: <a href="https://build.whamcloud.com/job/lustre-master/4445" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master/4445</a> - 4.18.0-477.15.1.el8_lustre.x86_64</p> <p>This started around 2023-07-21 +/- 7 days. It looks like the workqueue is somehow running <b>after</b> the journal is cleaned up, since the BUG is </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-object">int</span> jbd2_journal_destroy(journal_t *journal) { <span class="code-comment">/* Wait <span class="code-keyword">for</span> the commit thread to wake up and die. */</span> journal_kill_thread(journal); : } <span class="code-keyword">static</span> void journal_kill_thread(journal_t *journal) { journal-&gt;j_flags |= JBD2_UNMOUNT; : } <span class="code-keyword">static</span> <span class="code-object">int</span> start_this_handle(journal_t *journal, handle_t *handle, gfp_t gfp_mask) { : BUG_ON(journal-&gt;j_flags &amp; JBD2_UNMOUNT); : } </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_820 - trevis-28vm3 crashed during sanity test_820</p> LU-17332

sanity test_820: kernel BUG at fs/jbd2/transaction.c:378

Bug Critical Resolved Fixed Dongyang Li Maloo Mon, 4 Dec 2023 08:56:13 +0000 Wed, 3 Jan 2024 14:20:01 +0000 Wed, 3 Jan 2024 14:20:01 +0000 Lustre 2.16.0 Lustre 2.16.0 0 7 <p>I pushed patch <a href="https://review.whamcloud.com/53242" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/53242</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16032" title=" Truncate for large objects can lead to a thread hung" class="issue-link" data-issue-key="LU-16032"><del>LU-16032</del></a> osd-ldiskfs: track backlog of unlinked objects</tt>" to see if I can flush the workqueue at unmount time, but it doesn't seem to be quite correct. It seems to make the unmount worse, adding a crash in lu_env handling, possibly because the accounting is done in the OSD device instead of the superblock.</p> <p>Dongyang</p> <p>Could you please investigate?</p> <p>Thanks</p> <p>Peter</p> <p>The test failure referred by the ticket was on 2023-07-21 <a href="https://testing.whamcloud.com/test_sets/58a6b07c-fb1f-4a2d-ac3c-d7578d6b134f" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/58a6b07c-fb1f-4a2d-ac3c-d7578d6b134f</a><br/> And I believe it's because of <a href="https://jira.whamcloud.com/browse/LU-16982" title="Crash lustre after umount -d -f /mnt/lustre-mds4" class="issue-link" data-issue-key="LU-16982"><del>LU-16982</del></a>, the superblock update was scheduled after journal destroy, since the patch from <a href="https://jira.whamcloud.com/browse/LU-16982" title="Crash lustre after umount -d -f /mnt/lustre-mds4" class="issue-link" data-issue-key="LU-16982"><del>LU-16982</del></a> landed in Aug, we should not see this anymore. and I just did a grep on the trevis dump files to verify this.</p> <p>However I do find out there's a similar crash happening recently on el9 servers, the trace is like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 9096.356946] Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-ost2 [ 9097.522537] LDISKFS-fs (dm-9): unmounting filesystem. [ 9098.085442] ------------[ cut here ]------------ [ 9098.086099] kernel BUG at fs/jbd2/transaction.c:384! [ 9098.086804] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 9098.087460] CPU: 0 PID: 316553 Comm: kworker/0:2 Kdump: loaded Tainted: G OE -------- --- 5.14.0-284.30.1_lustre.el9.x86_64 #1 [ 9098.088869] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 9098.089558] Workqueue: events flush_stashed_error_work [ldiskfs] [ 9098.090478] RIP: 0010:start_this_handle+0x234/0x580 [jbd2] [ 9098.091187] Code: 40 48 89 44 24 48 41 83 79 0c 02 74 1f 0f 0b 48 89 ef e8 5f 1c 67 e7 48 89 ef e8 67 1e 67 e7 49 8b 06 a8 01 0f 84 77 fe ff ff &lt;0f&gt; 0b 4d 8d be 90 00 00 00 ba 02 00 00 00 48 8d 74 24 28 4c 89 ff [ 9098.093212] RSP: 0018:ffffb92607a37d90 EFLAGS: 00010202 [ 9098.093837] RAX: 0000000000000031 RBX: ffff92b1f5e6ba48 RCX: 00000000000000e0 [ 9098.094663] RDX: 0000000000008d40 RSI: ffffffffc06c3bf8 RDI: ffff92b1e6e3884c [ 9098.095493] RBP: ffff92b1e6e3884c R08: 0000000000000100 R09: 0000000000000000 [ 9098.096334] R10: ffff92b220ddf900 R11: 0000000000000106 R12: ffff92b220ddf900 [ 9098.097167] R13: 0000000000000000 R14: ffff92b1e6e38800 R15: 0000000000000001 [ 9098.097995] FS: 0000000000000000(0000) GS:ffff92b27cc00000(0000) knlGS:0000000000000000 [ 9098.098933] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 9098.099609] CR2: 000055c0889ae840 CR3: 000000008ed7a005 CR4: 00000000000606f0 [ 9098.100438] Call Trace: [ 9098.100782] &lt;TASK&gt; [ 9098.101098] ? kmem_cache_alloc+0x17d/0x300 [ 9098.101661] jbd2__journal_start+0xfb/0x200 [jbd2] [ 9098.102254] jbd2_journal_start+0x19/0x20 [jbd2] [ 9098.102866] flush_stashed_error_work+0x4a/0xf0 [ldiskfs] [ 9098.103543] process_one_work+0x1e5/0x3b0 [ 9098.104078] ? rescuer_thread+0x390/0x390 [ 9098.104578] worker_thread+0x50/0x3a0 [ 9098.105052] ? rescuer_thread+0x390/0x390 [ 9098.105549] kthread+0xd9/0x100 [ 9098.105978] ? kthread_complete_and_exit+0x20/0x20 [ 9098.106571] ret_from_fork+0x22/0x30 [ 9098.107085] &lt;/TASK&gt; </pre> </div></div> <p>The reason is <a href="https://jira.whamcloud.com/browse/LU-16982" title="Crash lustre after umount -d -f /mnt/lustre-mds4" class="issue-link" data-issue-key="LU-16982"><del>LU-16982</del></a> didn't fix the problem for el9 series, only the el8. I will prepare a patch for this.</p> <p>"Li Dongyang &lt;dongyangli@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53499" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53499</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17332" title="sanity test_820: kernel BUG at fs/jbd2/transaction.c:378" class="issue-link" data-issue-key="LU-17332"><del>LU-17332</del></a> ldiskfs: do no update superblock after journal destroy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 63c16ede8ea0075f670039bf087579226da8ca76</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53499/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53499/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17332" title="sanity test_820: kernel BUG at fs/jbd2/transaction.c:378" class="issue-link" data-issue-key="LU-17332"><del>LU-17332</del></a> ldiskfs: do no update superblock after journal destroy<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8cdfa1d50806ff5ef165909956201592e4f11b9b</p> <p>Landed for 2.16</p> Related LU-16032 Development Rank 1|i043pb: Rank (Obsolete) 9223372036854775807 Severity