https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14453 Lustre <p>This affects 2.14.0-RC3. Hit this a couple of times under I/O stress testing (</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[349195.535733] LDISKFS-fs error (device dm-0) in ldiskfs_free_blocks:5437: IO failure [349195.536702] LDISKFS-fs error (device dm-0) in ldiskfs_free_blocks:5437: IO failure [349195.563895] Aborting journal on device md131. [349195.564407] Aborting journal on device md131. [349195.564433] LDISKFS-fs error (device dm-0) in ldiskfs_orphan_add:3367: Journal has aborted [349195.564435] LDISKFS-fs error (device dm-0) in ldiskfs_orphan_del:3447: Journal has aborted [349195.564439] LDISKFS-fs error (device dm-0) in ldiskfs_do_update_inode:4840: Journal has aborted [349195.564442] LDISKFS-fs error (device dm-0) in ldiskfs_do_update_inode:4840: Journal has aborted [349195.564445] LDISKFS-fs error (device dm-0) in ldiskfs_do_update_inode:4840: Journal has aborted [349195.564483] LDISKFS-fs error (device dm-0) in ldiskfs_do_update_inode:4840: Journal has aborted [349195.564485] Kernel panic - not syncing: LDISKFS-fs (device dm-0): panic forced after error [349195.564486] LDISKFS-fs error (device dm-0) in ldiskfs_do_update_inode:4840: Journal has aborted [349195.564489] CPU: 18 PID: 25430 Comm: ll_ost00_576 Kdump: loaded Tainted: P OE ------------ 3.10.0-957.1.3957.1.3.x3.4.37.x86_64 #1 [349195.564490] Hardware name: Seagate Laguna Seca/Laguna Seca, BIOS v02.0040 06/29/2018 [349195.564491] LDISKFS-fs error (device dm-0) in ldiskfs_do_update_inode:4840: Journal has aborted [349195.564492] Call Trace: [349195.564500] [&lt;ffffffffb3763e41&gt;] dump_stack+0x19/0x1b [349195.564519] [&lt;ffffffffb375d550&gt;] panic+0xe8/0x21f [349195.564538] [&lt;ffffffffc1f1e156&gt;] ldiskfs_handle_error.part.191+0xa6/0xb0 [ldiskfs] [349195.564550] [&lt;ffffffffc1f1e7db&gt;] __ldiskfs_std_error+0x7b/0x100 [ldiskfs] [349195.564562] [&lt;ffffffffc1f3708a&gt;] ldiskfs_free_blocks+0xa1a/0xbb0 [ldiskfs] [349195.564570] [&lt;ffffffffc1eedef9&gt;] ldiskfs_ext_remove_space+0x8a9/0x1150 [ldiskfs] [349195.564576] [&lt;ffffffffc1f06525&gt;] ? ldiskfs_do_update_inode+0x525/0x810 [ldiskfs] [349195.564582] [&lt;ffffffffc1ef0670&gt;] ldiskfs_ext_truncate+0xb0/0xe0 [ldiskfs] [349195.564588] [&lt;ffffffffc1f0976a&gt;] ldiskfs_truncate+0x3da/0x430 [ldiskfs] [349195.564594] [&lt;ffffffffc1f0a4ea&gt;] ldiskfs_evict_inode+0x58a/0x630 [ldiskfs] [349195.564596] [&lt;ffffffffb325eac4&gt;] evict+0xb4/0x180 [349195.564597] [&lt;ffffffffb325f3cc&gt;] iput+0xfc/0x190 [349195.564606] [&lt;ffffffffc1f7f137&gt;] osd_object_delete+0x1e7/0x360 [osd_ldiskfs] [349195.564636] [&lt;ffffffffc1576ff8&gt;] lu_object_free.isra.27+0xb8/0x1c0 [obdclass] [349195.564653] [&lt;ffffffffc157a955&gt;] lu_object_put+0xa5/0x430 [obdclass] [349195.564661] [&lt;ffffffffc1dea40e&gt;] ofd_destroy_by_fid+0x20e/0x500 [ofd] [349195.564697] [&lt;ffffffffc183a820&gt;] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [349195.564720] [&lt;ffffffffc1836b80&gt;] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [349195.564725] [&lt;ffffffffc1ddc307&gt;] ofd_destroy_hdl+0x267/0x9f0 [ofd] [349195.564762] [&lt;ffffffffc18d472a&gt;] tgt_request_handle+0x7ea/0x1750 [ptlrpc] [349195.564816] [&lt;ffffffffc18ad241&gt;] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [349195.564843] [&lt;ffffffffc0e1702e&gt;] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [349195.564898] [&lt;ffffffffc18767a6&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [349195.564902] [&lt;ffffffffb30cec64&gt;] ? __wake_up+0x44/0x50 [349195.564939] [&lt;ffffffffc187b2fc&gt;] ptlrpc_main+0xb3c/0x14e0 [ptlrpc] [349195.564974] [&lt;ffffffffc187a7c0&gt;] ? ptlrpc_register_service+0xf90/0xf90 [ptlrpc] [349195.564976] [&lt;ffffffffb30c1c31&gt;] kthread+0xd1/0xe0 [349195.564977] [&lt;ffffffffb30c1b60&gt;] ? insert_kthread_work+0x40/0x40 [349195.564979] [&lt;ffffffffb3776c1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [349195.564980] [&lt;ffffffffb30c1b60&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>And an earlier hit on RC2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[59746.814376] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff932e1294c9c0] [phy 16627861504] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67042458] [59746.814380] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff9330c3183ea0] [phy 16627863552] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67344087] [59746.814383] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff932b18244888] [phy 16627865600] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67042515] [59746.814404] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff93206a2f1478] [phy 16627867648] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67042531] [59746.814409] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff932e12848c98] [phy 16627869696] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67042485] [59746.814416] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff932de984eaf8] [phy 16627871744] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67042535] [59746.814469] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff9330575f4618] [phy 16627873792] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67344057] [59746.814475] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff932fff6d21a0] [phy 16627875840] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67042505] [59746.814536] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff932de984f8f0] [phy 16627877888] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67042513] [59746.814539] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff933057701270] [phy 16627879936] [logic 2048] [len 2048] [free 2048] [error 1] [inode 67344093] [59870.004253] LDISKFS-fs warning: 39 callbacks suppressed [59870.011429] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff9329432eac98] [phy 16628195328] [logic 8192] [len 8192] [free 8192] [error 1] [inode 67042426] [59876.053222] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff932fff61bbc8] [phy 16628289536] [logic 4096] [len 4096] [free 4096] [error 1] [inode 67344041] [59876.153987] LDISKFS-fs warning (device dm-0): ldiskfs_mb_new_blocks:5077: Updating bitmap error: [err -5] [pa ffff93295856d6e8] [phy 16628322304] [logic 4096] [len 4096] [free 4096] [error 1] [inode 67042443] [59886.946880] LDISKFS-fs error (device dm-0) in ldiskfs_free_blocks:5437: IO failure [59887.691671] Aborting journal on device md129. [59887.890886] Kernel panic - not syncing: LDISKFS-fs (device dm-0): panic forced after error [59887.903674] CPU: 2 PID: 15660 Comm: ll_ost00_357 Kdump: loaded Tainted: P OE ------------ 3.10.0-957.1.3957.1.3.x3.4.37.x86_64 #1 [59887.919307] Hardware name: Seagate Laguna Seca/Laguna Seca, BIOS v02.0040 06/29/2018 [59887.928496] Call Trace: [59887.932396] [&lt;ffffffff97b63e41&gt;] dump_stack+0x19/0x1b [59887.938877] [&lt;ffffffff97b5d550&gt;] panic+0xe8/0x21f [59887.944972] [&lt;ffffffffc1cec156&gt;] ldiskfs_handle_error.part.191+0xa6/0xb0 [ldiskfs] [59887.953962] [&lt;ffffffffc1cec7db&gt;] __ldiskfs_std_error+0x7b/0x100 [ldiskfs] [59887.962318] [&lt;ffffffffc1d0508a&gt;] ldiskfs_free_blocks+0xa1a/0xbb0 [ldiskfs] [59887.970693] [&lt;ffffffff9761b591&gt;] ? __slab_free+0x81/0x2f0 [59887.977580] [&lt;ffffffffc1cbbef9&gt;] ldiskfs_ext_remove_space+0x8a9/0x1150 [ldiskfs] [59887.986704] [&lt;ffffffffc1cfa749&gt;] ? ldiskfs_es_free_extent+0x99/0x130 [ldiskfs] [59887.995452] [&lt;ffffffffc1cbe670&gt;] ldiskfs_ext_truncate+0xb0/0xe0 [ldiskfs] [59888.003741] [&lt;ffffffffc1cd776a&gt;] ldiskfs_truncate+0x3da/0x430 [ldiskfs] [59888.011910] [&lt;ffffffffc1cd84ea&gt;] ldiskfs_evict_inode+0x58a/0x630 [ldiskfs] [59888.020159] [&lt;ffffffff9765eac4&gt;] evict+0xb4/0x180 [59888.026205] [&lt;ffffffff9765f3cc&gt;] iput+0xfc/0x190 [59888.032015] [&lt;ffffffffc1d4d137&gt;] osd_object_delete+0x1e7/0x360 [osd_ldiskfs] [59888.040336] [&lt;ffffffffc1377ff8&gt;] lu_object_free.isra.27+0xb8/0x1c0 [obdclass] [59888.048763] [&lt;ffffffffc137b955&gt;] lu_object_put+0xa5/0x430 [obdclass] [59888.056395] [&lt;ffffffffc1bb840e&gt;] ofd_destroy_by_fid+0x20e/0x500 [ofd] [59888.064161] [&lt;ffffffffc163b820&gt;] ? ldlm_blocking_ast_nocheck+0x310/0x310 [ptlrpc] [59888.072993] [&lt;ffffffffc1637b80&gt;] ? ldlm_expired_completion_wait+0x2a0/0x2a0 [ptlrpc] [59888.081995] [&lt;ffffffffc1baa307&gt;] ofd_destroy_hdl+0x267/0x9f0 [ofd] [59888.089407] [&lt;ffffffffc16d572a&gt;] tgt_request_handle+0x7ea/0x1750 [ptlrpc] [59888.097512] [&lt;ffffffffc16ae241&gt;] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [59888.106229] [&lt;ffffffffc0fac02e&gt;] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [59888.114405] [&lt;ffffffffc16777a6&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [59888.123233] [&lt;ffffffff974cec64&gt;] ? __wake_up+0x44/0x50 [59888.129602] [&lt;ffffffffc167c2fc&gt;] ptlrpc_main+0xb3c/0x14e0 [ptlrpc] [59888.133296] LDISKFS-fs error (device dm-0) in ldiskfs_delete_entry:2838: Journal has aborted [59888.146602] [&lt;ffffffffc167b7c0&gt;] ? ptlrpc_register_service+0xf90/0xf90 [ptlrpc] [59888.155066] [&lt;ffffffff974c1c31&gt;] kthread+0xd1/0xe0 [59888.161030] [&lt;ffffffff974c1b60&gt;] ? insert_kthread_work+0x40/0x40 [59888.168206] [&lt;ffffffff97b76c1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [59888.175648] [&lt;ffffffff974c1b60&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> LU-14453

LDISKFS-fs error (device xxx) in ldiskfs_do_update_inode:4840: Journal has aborted

Bug Resolved Not a Bug WC Triage Shaun Tancheff Fri, 19 Feb 2021 13:43:41 +0000 Sat, 27 Feb 2021 13:46:22 +0000 Sat, 27 Feb 2021 13:46:22 +0000 0 6 <p>something went wrong with low-level IO:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [349195.535733] LDISKFS-fs error (device dm-0) in ldiskfs_free_blocks:5437: IO failure </pre> </div></div> <p>The second failure also starts with IO errors, so this looks like it is a storage problem. </p> <p>Right, we agree it looks like IO errors below Lustre. This storage HW runs 2.12.x OK and we're seeing this (and more related ldiskfs issues) on multiple systems without any indication of lower level IO errors. We're continuing our investigations. </p> <p>can you please attach full system log?</p> <p>This bug happens because of the original buggy version of <a href="https://jira.whamcloud.com/browse/LU-12988" title="improve mount time on huge ldiskfs filesystem" class="issue-link" data-issue-key="LU-12988"><del>LU-12988</del></a> prefetch code which was split to a separate RHEL 7.6 series by commit fc87b01 (<a href="https://jira.whamcloud.com/browse/LU-12477" title="Remove obsolete config checks" class="issue-link" data-issue-key="LU-12477"><del>LU-12477</del></a>) and never fixed by the updated patches in <a href="https://jira.whamcloud.com/browse/LU-12988" title="improve mount time on huge ldiskfs filesystem" class="issue-link" data-issue-key="LU-12988"><del>LU-12988</del></a>.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Author: James Simmons &lt;jsimmons@infradead.org&gt; Date: Thu Mar 5 20:40:47 2020 -0500 LU-12477 ldiskfs: remove obsolete ext4 patches </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> ldiskfs/kernel_patches/patches/rhel7.6# git blame ext4-mballoc-prefetch.patch ... fc87b01f96e (James Simmons 2020-03-05 20:40:47 -0500 27) @@ -524,7 +532,7 @@ ext4_read_block_bitmap(struct super_b fc87b01f96e (James Simmons 2020-03-05 20:40:47 -0500 28) struct buffer_head *bh; fc87b01f96e (James Simmons 2020-03-05 20:40:47 -0500 29) <span class="code-object">int</span> err; fc87b01f96e (James Simmons 2020-03-05 20:40:47 -0500 30) fc87b01f96e (James Simmons 2020-03-05 20:40:47 -0500 31) - bh = ext4_read_block_bitmap_nowait(sb, block_group); fc87b01f96e (James Simmons 2020-03-05 20:40:47 -0500 32) + bh = ext4_read_block_bitmap_nowait(sb, block_group, 1); fc87b01f96e (James Simmons 2020-03-05 20:40:47 -0500 33) <span class="code-keyword">if</span> (!bh) fc87b01f96e (James Simmons 2020-03-05 20:40:47 -0500 34) <span class="code-keyword">return</span> NULL; fc87b01f96e (James Simmons 2020-03-05 20:40:47 -0500 35) err = ext4_wait_block_bitmap(sb, block_group, bh); </pre> </div></div> <p>Thanks for the detailed explanation. </p> <p>Is that patch still in use for EL7.9 or is it only affecting EL7.6/7.7?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>, only RHEL 7.6 seems to be affected.</p> <p>Thanks for the analysis Andrew. It seems like this issue does not affect the community releases then</p> Related LU-12477 LU-12988 Development Rank 1|i01n3j: Rank (Obsolete) 9223372036854775807 Severity