https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2099 Lustre <p>Just hit this when testing with USE_OFD=yes</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 929.726136] Lustre: DEBUG MARKER: == sanity test 63a: Verify oig_wait interruption does not crash ========= 03:34:59 (1349595299) [ 964.438007] LustreError: 28767:0:(osd_io.c:107:osd_init_iobuf()) ASSERTION( iobuf-&gt;dr_elapsed_valid == 0 ) failed: [ 964.439525] LustreError: 28767:0:(osd_io.c:107:osd_init_iobuf()) LBUG [ 964.441022] Pid: 28767, comm: ll_ost_io00_004 [ 964.442579] [ 964.442580] Call Trace: [ 964.445464] [&lt;ffffffffa03f4915&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [ 964.446900] [&lt;ffffffffa03f4f27&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [ 964.448333] [&lt;ffffffffa0c41a82&gt;] osd_init_iobuf+0xd2/0xe0 [osd_ldiskfs] [ 964.449770] [&lt;ffffffffa0c44b0b&gt;] osd_write_prep+0x17b/0x420 [osd_ldiskfs] [ 964.451208] [&lt;ffffffffa0c43976&gt;] ? osd_bufs_get+0x206/0x390 [osd_ldiskfs] [ 964.452627] [&lt;ffffffffa0e26daa&gt;] ofd_preprw+0x7da/0x11c0 [ofd] [ 964.454077] [&lt;ffffffffa0781cf4&gt;] ? sptlrpc_svc_alloc_rs+0x74/0x2d0 [ptlrpc] [ 964.455474] [&lt;ffffffffa0d557fc&gt;] obd_preprw+0x12c/0x3d0 [ost] [ 964.456890] [&lt;ffffffffa0d5c932&gt;] ost_brw_write+0x882/0x15f0 [ost] [ 964.458284] [&lt;ffffffff8127a646&gt;] ? vsnprintf+0x2b6/0x5f0 [ 964.459696] [&lt;ffffffffa0754cbc&gt;] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] [ 964.461128] [&lt;ffffffffa0754e18&gt;] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] [ 964.462513] [&lt;ffffffffa0d621f0&gt;] ost_handle+0x3120/0x4550 [ost] [ 964.463899] [&lt;ffffffffa0401464&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [ 964.465329] [&lt;ffffffffa0762883&gt;] ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [ 964.466745] [&lt;ffffffffa03f566e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [ 964.468163] [&lt;ffffffffa075b571&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [ 964.469568] [&lt;ffffffff81057d60&gt;] ? default_wake_function+0x0/0x20 [ 964.470993] [&lt;ffffffffa076541a&gt;] ptlrpc_main+0xb9a/0x1960 [ptlrpc] [ 964.472399] [&lt;ffffffffa0764880&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [ 964.473806] [&lt;ffffffff8100c14a&gt;] child_rip+0xa/0x20 [ 964.475201] [&lt;ffffffffa0764880&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [ 964.476610] [&lt;ffffffffa0764880&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [ 964.477937] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 [ 964.479240] [ 964.481682] Kernel panic - not syncing: LBUG [ 964.482986] Pid: 28767, comm: ll_ost_io00_004 Not tainted 2.6.32-debug #6 [ 964.484296] Call Trace: [ 964.485627] [&lt;ffffffff814f75e4&gt;] ? panic+0xa0/0x168 [ 964.486964] [&lt;ffffffffa03f4f7b&gt;] ? lbug_with_loc+0x9b/0xb0 [libcfs] [ 964.488261] [&lt;ffffffffa0c41a82&gt;] ? osd_init_iobuf+0xd2/0xe0 [osd_ldiskfs] [ 964.489572] [&lt;ffffffffa0c44b0b&gt;] ? osd_write_prep+0x17b/0x420 [osd_ldiskfs] [ 964.490866] [&lt;ffffffffa0c43976&gt;] ? osd_bufs_get+0x206/0x390 [osd_ldiskfs] [ 964.492151] [&lt;ffffffffa0e26daa&gt;] ? ofd_preprw+0x7da/0x11c0 [ofd] [ 964.493512] [&lt;ffffffffa0781cf4&gt;] ? sptlrpc_svc_alloc_rs+0x74/0x2d0 [ptlrpc] [ 964.494817] [&lt;ffffffffa0d557fc&gt;] ? obd_preprw+0x12c/0x3d0 [ost] [ 964.496116] [&lt;ffffffffa0d5c932&gt;] ? ost_brw_write+0x882/0x15f0 [ost] [ 964.497432] [&lt;ffffffff8127a646&gt;] ? vsnprintf+0x2b6/0x5f0 [ 964.498774] [&lt;ffffffffa0754cbc&gt;] ? lustre_msg_get_version+0x8c/0x100 [ptlrpc] [ 964.500109] [&lt;ffffffffa0754e18&gt;] ? lustre_msg_check_version+0xe8/0x100 [ptlrpc] [ 964.501424] [&lt;ffffffffa0d621f0&gt;] ? ost_handle+0x3120/0x4550 [ost] [ 964.502716] [&lt;ffffffffa0401464&gt;] ? libcfs_id2str+0x74/0xb0 [libcfs] [ 964.504014] [&lt;ffffffffa0762883&gt;] ? ptlrpc_server_handle_request+0x463/0xe70 [ptlrpc] [ 964.505319] [&lt;ffffffffa03f566e&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [ 964.506593] [&lt;ffffffffa075b571&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [ 964.507817] [&lt;ffffffff81057d60&gt;] ? default_wake_function+0x0/0x20 [ 964.509104] [&lt;ffffffffa076541a&gt;] ? ptlrpc_main+0xb9a/0x1960 [ptlrpc] [ 964.510345] [&lt;ffffffffa0764880&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [ 964.511510] [&lt;ffffffff8100c14a&gt;] ? child_rip+0xa/0x20 [ 964.512795] [&lt;ffffffffa0764880&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [ 964.513993] [&lt;ffffffffa0764880&gt;] ? ptlrpc_main+0x0/0x1960 [ptlrpc] [ 964.515105] [&lt;ffffffff8100c140&gt;] ? child_rip+0x0/0x20 </pre> </div></div> <p>I have a crashdump for this instance too in case you need something from there.</p> single node, 4 cpu cores. lustre files in /tmp that is tmpfs LU-2099

osd_init_iobuf()) ASSERTION( iobuf->dr_elapsed_valid == 0 ) sanity test 63a

Bug Blocker Resolved Fixed Alex Zhuravlev Oleg Drokin Sun, 7 Oct 2012 03:46:35 +0000 Wed, 14 May 2014 22:32:41 +0000 Wed, 14 May 2014 15:36:47 +0000 Lustre 2.4.0 Lustre 2.5.0 Lustre 2.6.0 Lustre 2.6.0 Lustre 2.5.2 0 5 <p>Seems to be mostly the sane as ORI-258</p> <p>I just hit this one two more times on two different nodes, so this one is certainly a very real problem.</p> <p>do you have crash dumps for the case?</p> <p><a href="http://review.whamcloud.com/#change,4216" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4216</a></p> <p>I've yet to hit it again with the patch applied, though there was one strange problem hit in the same path once.</p> <p>from what I heard last time, no more hits happened with the patch.</p> <p>Seeing this issue again on latest master.</p> <p>loop: Write error at byte offset 243236864, length 4096.<br/> LustreError: 15444:0:(osd_io.c:71:__osd_init_iobuf()) ASSERTION( iobuf-&gt;dr_elapsed_valid == 0 ) failed: iobuf ffff880548d92c00, reqs 0, rw 1, line 1079<br/> LustreError: 15444:0:(osd_io.c:71:__osd_init_iobuf()) LBUG<br/> Pid: 15444, comm: ll_ost_io01_021</p> <p>Call Trace:<br/> <span class="error">&#91;&lt;ffffffffa09e8895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa09e8e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa054c8b8&gt;&#93;</span> __osd_init_iobuf+0x338/0x350 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa1232b83&gt;&#93;</span> ? lu_context_init+0xa3/0x240 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa0550a42&gt;&#93;</span> osd_write_commit+0x152/0x610 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa08821fa&gt;&#93;</span> ofd_commitrw_write+0x59a/0x11e0 <span class="error">&#91;ofd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa088336a&gt;&#93;</span> ofd_commitrw+0x52a/0x8c0 <span class="error">&#91;ofd&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa1210411&gt;&#93;</span> ? lprocfs_counter_add+0x151/0x1c0 <span class="error">&#91;obdclass&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa14c1c8d&gt;&#93;</span> obd_commitrw.clone.0+0x11d/0x390 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa14c8eae&gt;&#93;</span> tgt_brw_write+0xc7e/0x1530 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81059216&gt;&#93;</span> ? enqueue_task+0x66/0x80<br/> <span class="error">&#91;&lt;ffffffff8105571d&gt;&#93;</span> ? check_preempt_curr+0x6d/0x90<br/> <span class="error">&#91;&lt;ffffffff81065c5e&gt;&#93;</span> ? try_to_wake_up+0x24e/0x3e0<br/> <span class="error">&#91;&lt;ffffffffa14247c0&gt;&#93;</span> ? target_bulk_timeout+0x0/0xc0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa14c79ac&gt;&#93;</span> tgt_request_handle+0x23c/0xac0 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa146635c&gt;&#93;</span> ? lustre_msg_get_transno+0x8c/0x100 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffffa147698a&gt;&#93;</span> ptlrpc_main+0xd1a/0x1980 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff81068370&gt;&#93;</span> ? pick_next_task_fair+0xd0/0x130<br/> <span class="error">&#91;&lt;ffffffff81527e08&gt;&#93;</span> ? schedule+0x178/0x3b2<br/> <span class="error">&#91;&lt;ffffffffa1475c70&gt;&#93;</span> ? ptlrpc_main+0x0/0x1980 <span class="error">&#91;ptlrpc&#93;</span><br/> <span class="error">&#91;&lt;ffffffff8109aee6&gt;&#93;</span> kthread+0x96/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> <span class="error">&#91;&lt;ffffffff8109ae50&gt;&#93;</span> ? kthread+0x0/0xa0<br/> <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>I can't continue my testing since I'm keeping seeing this issue on 2.5 and 2.6</p> <p>could you grab full debug log for the case please? with the following patch:</p> <p>&#8212; a/lustre/osd-ldiskfs/osd_io.c<br/> +++ b/lustre/osd-ldiskfs/osd_io.c<br/> @@ -1139,7 +1139,9 @@ static int osd_write_commit(const struct lu_env *env, struct dt_object *dt,<br/> rc = osd_do_bio(osd, inode, iobuf);<br/> /* we don't do stats here as in read path because</p> <ul> <li>write is async: we'll do this in osd_put_bufs() */</li> </ul> <ul class="alternate" type="square"> <li>}<br/> + } else { + CERROR("NO IO: %d\n", rc); + }</li> </ul> <p> if (unlikely(rc != 0)) {<br/> /* if write fails, we should drop pages from the cache */</p> <p>Log and dmesg.</p> <p>I have a crash dump please let me know if you want it</p> <p>please, try with <a href="http://review.whamcloud.com/#/c/10305/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10305/</a><br/> I'm still doing local testing, but I'd appreciate your help given you can reproduce this easily.<br/> thanks in advance.</p> <p>it looks good in the initial testing</p> <p>Landed for 2.6</p> Related Development Rank 1|hzv59j: Rank (Obsolete) 4386 Severity