https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16245 Lustre <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 8753.247529] LustreError: 37772:0:(osd_io.c:79:__osd_init_iobuf()) ASSERTION( iobuf-&gt;dr_elapsed_valid == 0 ) failed: iobuf 000000006eba9531, reqs 0, rw 1, line 1633 [ 8753.262771] LustreError: 37772:0:(osd_io.c:79:__osd_init_iobuf()) LBUG [ 8753.269970] Pid: 37772, comm: mdt_io05_022 5.10.0-60.18.0.50.aarch64 #1 SMP Wed Oct 5 10:58:08 CST 2022 [ 8753.280021] Call Trace TBD: [ 8753.283505] Kernel panic - not syncing: LBUG [ 8753.288454] CPU: 59 PID: 37772 Comm: mdt_io05_022 Kdump: loaded Tainted: P           OE     5.10.0-60.18.0.50.aarch64 #1 [ 8753.299963] Hardware name: Huawei TaiShan 200 (Model 2280)/BC82AMDDA, BIOS 1.38 07/04/2020 [ 8753.308881] Call trace: [ 8753.312014]  dump_backtrace+0x0/0x1e0 [ 8753.316352]  show_stack+0x20/0x30 [ 8753.320347]  dump_stack+0xe0/0x148 [ 8753.324426]  panic+0x170/0x398 [ 8753.328188]  param_set_delay_minmax.isra.1+0x0/0xd0 [libcfs] [ 8753.334552]  __osd_init_iobuf+0x2e8/0x408 [osd_ldiskfs] [ 8753.340454]  osd_write_prep+0xec/0x330 [osd_ldiskfs] [ 8753.346149]  mdt_obd_preprw+0xaa0/0xc38 [mdt] [ 8753.351294]  tgt_brw_write+0x1208/0x2f30 [ptlrpc] [ 8753.351367]  tgt_handle_request0+0xd4/0x9b0 [ptlrpc] [ 8753.362369]  tgt_request_handle+0x7cc/0x1a30 [ptlrpc] [ 8753.368148]  ptlrpc_server_handle_request+0x3bc/0x1218 [ptlrpc] [ 8753.374791]  ptlrpc_main+0xdfc/0x16c8 [ptlrpc] [ 8753.379910]  kthread+0x130/0x138 [ 8753.383818]  ret_from_fork+0x10/0x18 [ 8753.388121] SMP: stopping secondary CPUs [ 8753.395179] Starting crashdump kernel... [ 8753.399781] Bye! </pre> </div></div> lustre servers: <br/> 10 nodes ,each node has kunpeng920 96core *2, memory 512GB,nvme 3.2T*4 <br/> centos 8.4.2105 <br/> kernel 5.10.0-60.18.0.50.aarch64 （openeuler 22.03 kernel） <br/> lustre 0c68b13a5eeb408862bad795aaf9a24a11a14b6a <br/> <br/> lustre clients: <br/> 10 nodes intel 6266C*2, memory 372GB <br/> centos 8.4.2105 <br/> kernel 4.18.0-372.9.1.el8.x86_64 <br/> <br/> IO500 tag:io500-sc21 <br/> LU-16245

__osd_init_iobuf()) ASSERTION( iobuf->dr_elapsed_valid == 0 )

Bug Critical Open Unresolved Jason Feng Jason Feng Tue, 18 Oct 2022 04:05:35 +0000 Tue, 29 Nov 2022 09:35:30 +0000 Lustre 2.15.1 Tue, 18 Oct 2022 00:00:00 +0000 0 3 <p>Do not modify dr_elapsed_valid if osd_fini_iobuf has been invoked.</p> <p>The initial value of dr_elapsed_valid is 0. When the I/O is complete, dio_complete_routine will set dr_elapsed_valid  to 1. Finally, dr_elapsed_valid is cleared in osd_fini_iobuf.In the I/O write process, wait_event is not called, and osd_fini_iobuf cannot be executed before dio_complete_routine. As a result, dr_elapsed_valid is not cleared and is asserted when it is used again.<br/> The initial value of dr_elapsed_valid is 0 and is changed to 2 in osd_fini_iobuf. The value of dr_elapsed_valid is changed to 1 only when the value of dr_elapsed_valid is 0 in dio_complete_routine. This avoids modification after finishing.</p> <p>"fengchunsong &lt;fengchunsong@huawei.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48905" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48905</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16245" title="__osd_init_iobuf()) ASSERTION( iobuf-&gt;dr_elapsed_valid == 0 )" class="issue-link" data-issue-key="LU-16245">LU-16245</a> osd-ldiskfs: prevent dr_elapsed_valid assertion<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5bc624a9c930f5dfd38b62eb661b706c418682e0</p> <p>I suspect this issue is similar to <a href="https://jira.whamcloud.com/browse/LU-12362" title="kernel warning &#39;do not call blocking ops when !TASK_RUNNING &#39; in ptlrpcd" class="issue-link" data-issue-key="LU-12362"><del>LU-12362</del></a>. Nested sleeping primitives might lead to an infinite wait, making osd_fini_iobuf() won't be called which causes this crash.</p> <p>See about the problem of nested sleeping primitives here: <a href="https://lwn.net/Articles/628628/" class="external-link" target="_blank" rel="nofollow noopener">https://lwn.net/Articles/628628/.</a> We might need to fix this issue like <a href="https://jira.whamcloud.com/browse/LU-12362" title="kernel warning &#39;do not call blocking ops when !TASK_RUNNING &#39; in ptlrpcd" class="issue-link" data-issue-key="LU-12362"><del>LU-12362</del></a>.</p> Related LU-16246 Development Epic/Theme Performance Rank 1|i0333z: Rank (Obsolete) 9223372036854775807 Severity