Whamcloud Community JIRA

Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us

9.4.14

940014

05-12-2023

[LU-11463] DOM: osc_extent_make_ready()) ASSERTION( last_oap_count > 0 ) failed https://jira.whamcloud.com/browse/LU-11463 Lustre <p>Separate DOM-related issue from the original one in <a href="https://jira.whamcloud.com/browse/LU-10407" title="osc_cache.c:1141:osc_extent_make_ready()) ASSERTION( last_oap_count > 0 ) failed: " class="issue-link" data-issue-key="LU-10407"><del>LU-10407</del></a> because there can be different reasons.</p> LU-11463

DOM: osc_extent_make_ready()) ASSERTION( last_oap_count > 0 ) failed

Bug Major Reopened Unresolved Mikhail Pershin Mikhail Pershin Wed, 3 Oct 2018 10:31:41 +0000 Sat, 23 Jan 2021 00:19:26 +0000 Upstream 0 5 <p>Copy of message from Oleg about this issue:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[35616.314978] LustreError: 11162:0:(osc_cache.c:1141:osc_extent_make_ready()) ASSERTION( last_oap_count > 0 ) failed: [35616.322765] LustreError: 11162:0:(osc_cache.c:1141:osc_extent_make_ready()) LBUG [35616.326801] Pid: 11162, comm: ldlm_bl_05 3.10.0-7.5-debug #1 SMP Sun Jun 3 13:35:38 EDT 2018 [35616.329285] Call Trace: [35616.330443] [<ffffffffa01eb7dc>] libcfs_call_trace+0x8c/0xc0 [libcfs] [35616.331870] [<ffffffffa01eb88c>] lbug_with_loc+0x4c/0xa0 [libcfs] [35616.334288] [<ffffffffa0d3eda6>] osc_extent_make_ready+0x936/0xe70 [osc] [35616.335640] [<ffffffffa0d45ab3>] osc_cache_writeback_range+0x4f3/0x1260 [osc] [35616.337641] [<ffffffffa0e0bac7>] mdc_lock_flush+0x2e7/0x3f0 [mdc] [35616.339070] [<ffffffffa0e0bfb4>] mdc_ldlm_blocking_ast+0x2f4/0x3f0 [mdc] [35616.343073] [<ffffffffa0b01bd4>] ldlm_cancel_callback+0x84/0x320 [ptlrpc] [35616.344357] [<ffffffffa0b189b0>] ldlm_cli_cancel_local+0xa0/0x420 [ptlrpc] [35616.345719] [<ffffffffa0b1e7e7>] ldlm_cli_cancel+0x157/0x620 [ptlrpc] [35616.347470] [<ffffffffa0e0be4a>] mdc_ldlm_blocking_ast+0x18a/0x3f0 [mdc] [35616.348765] [<ffffffffa0b2a67f>] ldlm_handle_bl_callback+0xff/0x530 [ptlrpc] [35616.351095] [<ffffffffa0b2afb1>] ldlm_bl_thread_main+0x501/0x680 [ptlrpc] [35616.352495] [<ffffffff810ae864>] kthread+0xe4/0xf0 [35616.353757] [<ffffffff81783777>] ret_from_fork_nospec_end+0x0/0x39 </pre> </div></div> <p>I can easily reproduce this with the linux lustre client</p> <p>James, this issue wasn't seen for a long time, looks like client is missing some related fixes. Where can I check its commit history?</p> <p>Currently I have a local kernel tree that is up to Lustre 2.12.0. Do you know patches fixed this issue?</p> <p>One of the patches that landed during 2.13 fixed this issue. We can reopen if it shows up again.</p> <p>Sorry but add direct I/O support to the linux client exposes this problem again. I can easily reproduce it with sanity.sh test </p> <p>James, could you describe how to reproduce that, is that happening in some particular test or occasionally? Any specific parameters for testing?</p> <p>James, if possible could you please point me the patch that fixed it initially. I also faced this while doing testing locally of my own patch (<a href="https://review.whamcloud.com/#/c/9275/)" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/9275/)</a> test 150b triggered it. I want to get hints what and how it got fixed. This is on the latest master : 742897a <a href="https://jira.whamcloud.com/browse/LU-13274" title="Building againt lustreapi using -std=c99" class="issue-link" data-issue-key="LU-13274"><del>LU-13274</del></a> uapi: make lnet UAPI headers C99 compliant</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 6559.142453] LustreError: 18592:0:(osc_cache.c:1124:osc_extent_make_ready()) ASSERTION( last_oap_count > 0 ) failed: [ 6559.142454] LustreError: 18592:0:(osc_cache.c:1124:osc_extent_make_ready()) LBUG [ 6559.142456] Pid: 18592, comm: ptlrpcd_00_01 3.10.0-957.el7_lustre.x86_64 #1 SMP Fri Dec 21 21:49:33 UTC 2018 [ 6559.142456] Call Trace: [ 6559.142477] [<ffffffffc0475e4c>] libcfs_call_trace+0x8c/0xc0 [libcfs] [ 6559.142481] [<ffffffffc0475efc>] lbug_with_loc+0x4c/0xa0 [libcfs] [ 6559.142490] [<ffffffffc0b6334c>] osc_extent_make_ready+0xb4c/0xe40 [osc] [ 6559.142496] [<ffffffffc0b662eb>] osc_io_unplug0+0xe9b/0x18b0 [osc] [ 6559.142501] [<ffffffffc0b3f773>] brw_queue_work+0x33/0xd0 [osc] [ 6559.142537] [<ffffffffc08f27fa>] work_interpreter+0x3a/0xf0 [ptlrpc] [ 6559.142561] [<ffffffffc08fb220>] ptlrpc_check_set+0x510/0x1ed0 [ptlrpc] [ 6559.142587] [<ffffffffc0929e0b>] ptlrpcd_check+0x4ab/0x590 [ptlrpc] [ 6559.142611] [<ffffffffc092a0a0>] ptlrpcd+0x1b0/0x6a0 [ptlrpc] [ 6559.142614] [<ffffffff876c1c31>] kthread+0xd1/0xe0 [ 6559.142617] [<ffffffff87d74c37>] ret_from_fork_nospec_end+0x0/0x39 [ 6559.142631] [<ffffffffffffffff>] 0xffffffffffffffff [ 6559.142631] Kernel panic - not syncing: LBUG [ 6559.142634] CPU: 1 PID: 18592 Comm: ptlrpcd_00_01 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.el7_lustre.x86_64 #1 [ 6559.142635] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 6559.142635] Call Trace: [ 6559.142639] [<ffffffff87d61dc1>] dump_stack+0x19/0x1b [ 6559.142641] [<ffffffff87d5b4d0>] panic+0xe8/0x21f [ 6559.142646] [<ffffffffc0475f4b>] lbug_with_loc+0x9b/0xa0 [libcfs] [ 6559.142653] [<ffffffffc0b6334c>] osc_extent_make_ready+0xb4c/0xe40 [osc] [ 6559.142672] [<ffffffffc0b662eb>] osc_io_unplug0+0xe9b/0x18b0 [osc] [ 6559.142677] [<ffffffffc0b3f773>] brw_queue_work+0x33/0xd0 [osc] [ 6559.142699] [<ffffffffc08f27fa>] work_interpreter+0x3a/0xf0 [ptlrpc] [ 6559.142721] [<ffffffffc08fb220>] ptlrpc_check_set+0x510/0x1ed0 [ptlrpc] [ 6559.142746] [<ffffffffc0929e0b>] ptlrpcd_check+0x4ab/0x590 [ptlrpc] [ 6559.142769] [<ffffffffc092a0a0>] ptlrpcd+0x1b0/0x6a0 [ptlrpc] [ 6559.142772] [<ffffffff8762a59e>] ? __switch_to+0xce/0x580 [ 6559.142773] [<ffffffff876c2d00>] ? wake_up_atomic_t+0x30/0x30 [ 6559.142796] [<ffffffffc0929ef0>] ? ptlrpcd_check+0x590/0x590 [ptlrpc] [ 6559.142798] [<ffffffff876c1c31>] kthread+0xd1/0xe0 [ 6559.142799] [<ffffffff876c1b60>] ? insert_kthread_work+0x40/0x40 [ 6559.142801] [<ffffffff87d74c37>] ret_from_fork_nospec_begin+0x21/0x21 [ 6559.142802] [<ffffffff876c1b60>] ? insert_kthread_work+0x40/0x40</pre> </div></div> <p><a href="https://testing.whamcloud.com/test_sessions/0f2a364a-716a-4eec-9786-3fa81b3143c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/0f2a364a-716a-4eec-9786-3fa81b3143c2</a></p> <p>I can easily reproduce it with the Linux client (<a href="https://github.com/jasimmons1973/lustre)." class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jasimmons1973/lustre).</a> If you run sanity test 398b it crashes every time. Also sanity-flr.sh test 200.</p> <p>Please note that Alex report and James comment is not about DoM as ticket says, I assume <a href="https://jira.whamcloud.com/browse/LU-10407" title="osc_cache.c:1141:osc_extent_make_ready()) ASSERTION( last_oap_count > 0 ) failed: " class="issue-link" data-issue-key="LU-10407"><del>LU-10407</del></a> is proper place to report non-DOM issue of this kind</p> <p>I'm stilling seeing failures with the Linux client. It might be specific to the Linux version so please keep this ticket open.</p> Related LU-10407 LU-12511 LU-14326 LU-13992 Development Rank 1|i003gv: Rank (Obsolete) 9223372036854775807 Severity