https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14402 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 6733.016726] perf: interrupt took too long (3163 &gt; 3126), lowering kernel.perf_event_max_sample_rate to 63000 [ 6953.568157] LustreError: 29459:0:(osd_io.c:1558:osd_write_commit()) ASSERTION( !PageDirty(lnb[i].lnb_page) ) failed: [ 6953.579769] LustreError: 29459:0:(osd_io.c:1558:osd_write_commit()) LBUG [ 6953.587203] Pid: 29459, comm: ll_ost_io00_523 3.10.0-957.1.3957.1.3.x3.4.37.x86_64 #1 SMP Mon Jan 13 18:26:28 PST 2020 [ 6953.598541] Call Trace: [ 6953.601529] [&lt;ffffffffc11e862c&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [ 6953.608765] [&lt;ffffffffc11e894c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [ 6953.615539] [&lt;ffffffffc1dffcac&gt;] osd_write_commit+0x52c/0x870 [osd_ldiskfs] [ 6953.623529] [&lt;ffffffffc1c24769&gt;] ofd_commitrw_write+0xde9/0x1480 [ofd] [ 6953.630806] [&lt;ffffffffc1c2861d&gt;] ofd_commitrw+0x2ad/0x9a0 [ofd] [ 6953.637512] [&lt;ffffffffc171dac9&gt;] tgt_brw_write+0xfd9/0x1cc0 [ptlrpc] [ 6953.644747] [&lt;ffffffffc1719c4a&gt;] tgt_request_handle+0x7ea/0x1750 [ptlrpc] [ 6953.652474] [&lt;ffffffffc16bd136&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [ 6953.661074] [&lt;ffffffffc16c1c8c&gt;] ptlrpc_main+0xb3c/0x14e0 [ptlrpc] [ 6953.668522] [&lt;ffffffffba6c1c31&gt;] kthread+0xd1/0xe0 [ 6953.674378] [&lt;ffffffffbad76c1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [ 6953.681517] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [ 6953.687422] Kernel panic - not syncing: LBUG [ 6953.692298] CPU: 11 PID: 29459 Comm: ll_ost_io00_523 Kdump: loaded Tainted: P OE ------------ 3.10.0-957.1.3957.1.3.x3.4.37.x86_64 #1 [ 6953.706675] Hardware name: Seagate Laguna Seca/Laguna Seca, BIOS v02.0040 06/29/2018 [ 6953.715123] Call Trace: [ 6953.718301] [&lt;ffffffffbad63e41&gt;] dump_stack+0x19/0x1b [ 6953.724122] [&lt;ffffffffbad5d550&gt;] panic+0xe8/0x21f [ 6953.729700] [&lt;ffffffffc11e899b&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [ 6953.736628] [&lt;ffffffffc1dffcac&gt;] osd_write_commit+0x52c/0x870 [osd_ldiskfs] [ 6953.744395] [&lt;ffffffffc1c24769&gt;] ofd_commitrw_write+0xde9/0x1480 [ofd] [ 6953.751678] [&lt;ffffffffc1c2861d&gt;] ofd_commitrw+0x2ad/0x9a0 [ofd] [ 6953.758374] [&lt;ffffffffc171dac9&gt;] tgt_brw_write+0xfd9/0x1cc0 [ptlrpc] [ 6953.765463] [&lt;ffffffffba6db748&gt;] ? __enqueue_entity+0x78/0x80 [ 6953.771941] [&lt;ffffffffba6e236f&gt;] ? enqueue_entity+0x2ef/0xbe0 [ 6953.778575] [&lt;ffffffffc16d6d97&gt;] ? __req_capsule_get+0x427/0x6b0 [ptlrpc] [ 6953.786210] [&lt;ffffffffc1719c4a&gt;] tgt_request_handle+0x7ea/0x1750 [ptlrpc] [ 6953.793817] [&lt;ffffffffc16f3bc1&gt;] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc] [ 6953.802150] [&lt;ffffffffc11e502e&gt;] ? ktime_get_real_seconds+0xe/0x10 [libcfs] [ 6953.809942] [&lt;ffffffffc16bd136&gt;] ptlrpc_server_handle_request+0x256/0xb10 [ptlrpc] [ 6953.818313] [&lt;ffffffffba6cec64&gt;] ? __wake_up+0x44/0x50 [ 6953.824396] [&lt;ffffffffc16c1c8c&gt;] ptlrpc_main+0xb3c/0x14e0 [ptlrpc] [ 6953.831440] [&lt;ffffffffc16c1150&gt;] ? ptlrpc_register_service+0xf90/0xf90 [ptlrpc] [ 6953.839649] [&lt;ffffffffba6c1c31&gt;] kthread+0xd1/0xe0 [ 6953.845394] [&lt;ffffffffba6c1b60&gt;] ? insert_kthread_work+0x40/0x40 [ 6953.852303] [&lt;ffffffffbad76c1d&gt;] ret_from_fork_nospec_begin+0x7/0x21 [ 6953.859587] [&lt;ffffffffba6c1b60&gt;] ? insert_kthread_work+0x40/0x40 (END) </pre> </div></div> LU-14402

LBUG: osd_write_commit() ASSERTION( !PageDirty(lnb[i].lnb_page) ) failed

Bug Major Resolved Cannot Reproduce WC Triage Shaun Tancheff Thu, 4 Feb 2021 13:18:34 +0000 Sat, 23 Oct 2021 01:34:09 +0000 Sat, 23 Oct 2021 01:34:09 +0000 Lustre 2.14.0 0 4 <p>As of: v2_14_54-52-g1887169365</p> <p>I am still hitting this LBUG.</p> <p>Ex:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Sep 19 09:21:52 snx11922n005 kernel: LustreError: 24267:0:(osd_io.c:1608:osd_write_commit()) ASSERTION( !PageDirty(lnb[i].lnb_page) ) failed: Sep 19 09:21:52 snx11922n005 kernel: LustreError: 24267:0:(osd_io.c:1608:osd_write_commit()) LBUG Sep 19 09:21:52 snx11922n005 kernel: Pid: 24267, comm: ll_ost_io00_713 3.10.0-957.1.3957.1.3.x3.4.37.x86_64 #1 SMP Mon Jan 13 18:26:28 PST 2020 Sep 19 09:21:52 snx11922n005 kernel: IEC: 026000003: LASSERT: { "pid": "24267", "ext_pid": "0", "filename": "osd_io.c", "line": "1608", "func_name": "osd_write_commit", "assert_info": "( !PageDirty(lnb[i].lnb_page) ) failed: " } Sep 19 09:21:52 snx11922n005 kernel: IEC: 026000004: LBUG: { "pid": "24267", "ext_pid": "0", "filename": "osd_io.c", "line": "1608", "func_name": "osd_write_commit" } Sep 19 09:21:52 snx11922n005 kernel: Call Trace: Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] libcfs_call_trace+0x90/0xf0 [libcfs] Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] osd_write_commit+0x52c/0x880 [osd_ldiskfs] Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] ofd_commitrw_write+0xef9/0x15d0 [ofd] Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] ofd_commitrw+0x335/0x9f0 [ofd] Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] tgt_brw_write+0x176a/0x2310 [ptlrpc] Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] tgt_request_handle+0x823/0x1850 [ptlrpc] Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] ptlrpc_server_handle_request+0x253/0xb10 [ptlrpc] Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] ptlrpc_main+0xbf4/0x15e0 [ptlrpc] Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] kthread+0xd1/0xe0 Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] ret_from_fork_nospec_begin+0x7/0x21 Sep 19 09:21:52 snx11922n005 kernel: [&lt;0&gt;] 0xfffffffffffffffe Sep 19 09:21:52 snx11922n005 kernel: Kernel panic - not syncing: LBUG </pre> </div></div> <p>Also kindly note that this includes the <a href="https://jira.whamcloud.com/browse/LU-14640" title="ASSERTION( !PageDirty(lnb[i].lnb_page) in osd_write_commit()" class="issue-link" data-issue-key="LU-14640"><del>LU-14640</del></a> fix <a href="https://review.whamcloud.com/#/c/43462/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/43462/</a></p> <p>"Shaun Tancheff &lt;shaun.tancheff@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45086" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45086</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14402" title="LBUG: osd_write_commit() ASSERTION( !PageDirty(lnb[i].lnb_page) ) failed" class="issue-link" data-issue-key="LU-14402"><del>LU-14402</del></a> osd-ldiskfs: Page cache pages dirtied in writeback<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3d35690a18d558be68e1701c71f9f02901125cf8</p> <p>"Shaun Tancheff &lt;shaun.tancheff@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45087" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45087</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14402" title="LBUG: osd_write_commit() ASSERTION( !PageDirty(lnb[i].lnb_page) ) failed" class="issue-link" data-issue-key="LU-14402"><del>LU-14402</del></a> osd-ldiskfs: disable pagecache bypass feature<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 060e753054494e44eb15bef9695d9038ba834a4e</p> <p><del>Shaun, this same <tt>ASSERTION()</tt> was reported fixed via patch <a href="https://review.whamcloud.com/43462" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/43462</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-14640" title="ASSERTION( !PageDirty(lnb[i].lnb_page) in osd_write_commit()" class="issue-link" data-issue-key="LU-14640"><del>LU-14640</del></a> osd: ASSERTION(!PageDirty(lnb<span class="error">&#91;i&#93;</span>.lnb_page)</tt>".</del></p> <p><del>The problem wasn't so much the uncached IO patches that you are reverting as that <tt>fallocate()</tt> wasn't cleaning up partially-zeroed pages in the cache.</del></p> <p>Ah, I see your addendum that this patch is included in your recent failure. Do you have any details of your workload that is triggering this error? We've been running with the uncached IO patches for many months without any similar reports in the field, so there must be something specific in your workload that is triggering it.</p> <p>The crash hits quite early (30 minutes or so) in our io-stress suite. The suite is large with bits of unaligned I/O mixed in doing to usual aio/dio, mmap, ior, and ltp tests.</p> <p>As of 14d07b6237 this did not reproduce.</p> <p>The range v2_14_55-52..14d07b6237 contains 87 patches, but if this is restricted to changes to <tt>lustre/osd-ldiskfs</tt> and <tt>lustre/ofd</tt> the list is more manageable:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@centos7 lustre-copy]# git log --oneline v2_14_55~$((98-52))..14d07b6237 lustre/osd-ldiskfs lustre/ofd 5daf86607877 LU-12268 osd: BUG_ON for IAM corruption 882a9f784de2 LU-14927 scrub: create shared scrub_needs_check() function. 0daeebcbdc4e LU-14797 nodemap: map project id bbfdc7c1670c LU-14739 quota: fix quota with root squash enabled bb5d81ea9550 LU-14543 target: prevent overflowing of tgd-&gt;tgd_tot_granted da1d93513fdf LU-14475 log: Rewrite some log messages 2a24b6ec67da LU-14734 ldiskfs: improve message for large_dir 7fdd664b3518 LU-14895 osd-ldiskfs: combine checksum functions c18d5d892b62 LU-14889 lproc: Add server checksum_type </pre> </div></div> <p>The patch <a href="https://review.whamcloud.com/45072" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45072</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12268" title="LDISKFS-fs error: ldiskfs_find_dest_de:2066: bad entry in directory: rec_len is smaller than minimal - offset=0( 0), inode=201, rec_len=0, name_len=0" class="issue-link" data-issue-key="LU-12268"><del>LU-12268</del></a> osd: BUG_ON for IAM corruption</tt>" which might "prevent" memory corruption in some cases, but it would trigger <tt>BUG_ON()</tt> instead, because the patch doesn't actually fix the core problem.</p> <p>Other than that, I don't really have any concrete suggestions other than to try bisect, if you want to figure out which patch fixed the problem.</p> Related LU-14640 Development Rank 1|i01liv: Rank (Obsolete) 9223372036854775807 Severity