https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5003 Lustre <p>This issue was created by maloo for wangdi &lt;di.wang@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="http://maloo.whamcloud.com/test_sets/d021bd8e-d2a0-11e3-a102-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">http://maloo.whamcloud.com/test_sets/d021bd8e-d2a0-11e3-a102-52540035b04c</a>.</p> <p>The sub-test test_302 failed with the following error:</p> <blockquote> <p>test failed to respond and timed out</p> <p>23:43:18:LustreError: 11-0: lustre-MDT0000-osp-MDT0001: Communicating with 10.1.4.101@tcp, operation obd_ping failed with -107.<br/> 23:43:18:LustreError: Skipped 6 previous similar messages<br/> 23:43:18:Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 10.1.4.101@tcp) was lost; in progress operations using this service will wait for recovery to complete<br/> 23:43:18:Lustre: Skipped 5 previous similar messages<br/> 23:43:18:LustreError: 166-1: MGC10.1.4.101@tcp: Connection to MGS (at 10.1.4.101@tcp) was lost; in progress operations using this service will fail<br/> 23:43:18:LustreError: 31202:0:(llog.c:407:llog_process_thread()) llog found corrupted<br/> 23:43:18:LustreError: 31202:0:(llog.c:870:llog_write()) ASSERTION( loghandle-&gt;lgh_obj != ((void *)0) ) failed: <br/> 23:43:18:LustreError: 31202:0:(llog.c:870:llog_write()) LBUG<br/> 23:43:18:Pid: 31202, comm: llog_process_th<br/> 23:43:18:<br/> 23:43:18:Call Trace:<br/> 23:43:18: <span class="error">&#91;&lt;ffffffffa0483895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> 23:43:18: <span class="error">&#91;&lt;ffffffffa0483e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> 23:43:18: <span class="error">&#91;&lt;ffffffffa059dd42&gt;&#93;</span> llog_write+0x352/0x420 <span class="error">&#91;obdclass&#93;</span><br/> 23:43:18: <span class="error">&#91;&lt;ffffffffa04945a1&gt;&#93;</span> ? libcfs_debug_msg+0x41/0x50 <span class="error">&#91;libcfs&#93;</span><br/> 23:43:18: <span class="error">&#91;&lt;ffffffffa059defc&gt;&#93;</span> llog_cancel_rec+0xbc/0x6a0 <span class="error">&#91;obdclass&#93;</span><br/> 23:43:18: <span class="error">&#91;&lt;ffffffffa059f073&gt;&#93;</span> llog_process_thread+0xb93/0xdc0 <span class="error">&#91;obdclass&#93;</span><br/> 23:43:18: <span class="error">&#91;&lt;ffffffffa05ed9cf&gt;&#93;</span> ? keys_fill+0x6f/0x190 <span class="error">&#91;obdclass&#93;</span><br/> 23:43:18: <span class="error">&#91;&lt;ffffffffa059f2e5&gt;&#93;</span> llog_process_thread_daemonize+0x45/0x70 <span class="error">&#91;obdclass&#93;</span><br/> 23:43:18: <span class="error">&#91;&lt;ffffffffa059f2a0&gt;&#93;</span> ? llog_process_thread_daemonize+0x0/0x70 <span class="error">&#91;obdclass&#93;</span><br/> 23:43:18: <span class="error">&#91;&lt;ffffffff8109aee6&gt;&#93;</span> kthread+0x96/0xa0<br/> 23:43:18: <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> 23:43:18: <span class="error">&#91;&lt;ffffffff8109ae50&gt;&#93;</span> ? kthread+0x0/0xa0<br/> 23:43:18: <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> 23:43:18:<br/> 23:43:18:Kernel panic - not syncing: LBUG</p></blockquote> <p>Info required for matching: sanity-hsm 302</p> LU-5003

sanity-hsm test_302 llog_write()) ASSERTION( loghandle->lgh_obj != ((void *)0)

Bug Critical Resolved Fixed James Nunez Maloo patch Sat, 3 May 2014 17:03:18 +0000 Mon, 6 Oct 2014 11:57:32 +0000 Mon, 6 Oct 2014 11:57:32 +0000 Lustre 2.7.0 0 11 <p>I saw the same issue while testing another patch, it happens in test_301. I tend to think that the reason might be commit <a href="http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=6760f383f6cd0964e70250723b9d24bce13b5318" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/?p=fs/lustre-release.git;a=commit;h=6760f383f6cd0964e70250723b9d24bce13b5318</a><br/> it changed the plan llog deletion recognition logic so I suppose llog was deleted during cancel and is being updated after that. This is just quick idea and area to check first.</p> <p>Lustre build: <a href="http://build.whamcloud.com/job/lustre-master/2065/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/2065/</a></p> <p>conf-sanity test 57a also hit the same failure:<br/> <a href="https://maloo.whamcloud.com/test_sets/6b7ba1ac-e5e1-11e3-87f3-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/6b7ba1ac-e5e1-11e3-87f3-52540035b04c</a></p> <p>Another failure instance:</p> <p><a href="https://testing.hpdd.intel.com/test_sets/8c42ce96-085b-11e4-8b2d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8c42ce96-085b-11e4-8b2d-5254006e85c2</a></p> <p>Another:</p> <p><a href="https://testing.hpdd.intel.com/test_logs/ef9a8e1e-27e8-11e4-8e75-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_logs/ef9a8e1e-27e8-11e4-8e75-5254006e85c2</a></p> <p><a href="https://testing.hpdd.intel.com/test_sets/6d49dfa6-3e26-11e4-a0a3-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/6d49dfa6-3e26-11e4-a0a3-5254006e85c2</a></p> <p>I have a crash looking exactly the same at replay-vbr test_4k , the llog being processed is a config llog.<br/> From a crashdump. the loghandle appears to be a client one:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lgh_hdr = 0xffff88005a8d0000, lgh_obj = 0x0 &lt;per_cpu__irq_stack_union&gt;, lgh_last_idx = 249, lgh_cur_idx = 0, lgh_cur_offset = 0, lgh_ctxt = 0xffff88005b433e40, ... lgh_name = 0x0 &lt;per_cpu__irq_stack_union&gt;, private_data = 0x0 &lt;per_cpu__irq_stack_union&gt;, lgh_logops = 0xffffffffa078f260 &lt;llog_client_ops&gt;, lgh_refcount = { counter = 1 } </pre> </div></div> <p>lgh_obj = 0x0 is what caused llog_write() to LBUG and lgh_logops = 0xffffffffa078f260 &lt;llog_client_ops&gt; indicates that the llog_handle is a client one.</p> <p>patch <a href="http://review.whamcloud.com/#/c/11955/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/11955/</a></p> <p>Seems it is <font color="red">not</font> right to compare error code with -EIO, llog block read functions returns -EINVAL if llog fle format check fail:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> <span class="code-object">int</span> llog_osd_next_block(<span class="code-keyword">const</span> struct lu_env *env, struct llog_handle *loghandle, <span class="code-object">int</span> *cur_idx, <span class="code-object">int</span> next_idx, __u64 *cur_offset, void *buf, <span class="code-object">int</span> len) { struct llog_thread_info *lgi = llog_info(env); ... <span class="code-keyword">if</span> (rc &lt; sizeof(*tail)) { CERROR(<span class="code-quote">"%s: invalid llog block at log id "</span>DOSTID<span class="code-quote">"/%u "</span> <span class="code-quote">"offset "</span>LPU64<span class="code-quote">"\n"</span>, o-&gt;do_lu.lo_dev-&gt;ld_obd-&gt;obd_name, POSTID(&amp;loghandle-&gt;lgh_id.lgl_oi), loghandle-&gt;lgh_id.lgl_ogen, *cur_offset); GOTO(out, rc = -EINVAL); } ... <span class="code-comment">/* <span class="code-keyword">this</span> shouldn't happen */</span> <span class="code-keyword">if</span> (tail-&gt;lrt_index == 0) { CERROR(<span class="code-quote">"%s: invalid llog tail at log id "</span>DOSTID<span class="code-quote">"/%u "</span> <span class="code-quote">"offset "</span>LPU64<span class="code-quote">"\n"</span>, o-&gt;do_lu.lo_dev-&gt;ld_obd-&gt;obd_name, POSTID(&amp;loghandle-&gt;lgh_id.lgl_oi), loghandle-&gt;lgh_id.lgl_ogen, *cur_offset); GOTO(out, rc = -EINVAL); } ... </pre> </div></div> <p>EIO is likely a temporary error (network, disk access, ...), EINVAL &#8211; points to a llog corruption.<br/> would it be better to change <tt>rc == -EIO</tt> check to <tt>rc == -EINVAL &amp;&amp; ...</tt> ?</p> <p>Thanks Zam!</p> <p>James</p> <p>Could you please review this patch?</p> <p>Thanks</p> <p>Peter</p> <p>Landed for 2.7</p> Duplicate LU-5019 Development Rank 1|hzwlrj: Rank (Obsolete) 13852 Severity