https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3278 Lustre <p>Running recovery-small.sh hit this very fast after test start.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 838.403817] Lustre: DEBUG MARKER: == replay-single test 2b: touch == 00:56:23 (1367729783) [ 839.177827] LustreError: 6825:0:(osd_handler.c:1119:osd_ro()) *** setting lustre-MDT0000 read-only *** [ 839.178349] Turning device loop0 (0x700000) read-only [ 839.231722] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 839.261830] Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 [ 839.589175] Lustre: Failing over lustre-MDT0000 [ 839.716258] LustreError: 6547:0:(osd_internal.h:747:osd_fid2oi()) ASSERTION( osd-&gt;od_oi_table != ((void *)0) &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed: [ 839.716797] LustreError: 6547:0:(osd_internal.h:747:osd_fid2oi()) LBUG [ 839.717086] Pid: 6547, comm: ll_mgs_0002 [ 839.717336] [ 839.717337] Call Trace: [ 839.717752] [&lt;ffffffffa04018a5&gt;] libcfs_debug_dumpstack+0x55/0x80 [libcfs] [ 839.718105] [&lt;ffffffffa0401ea7&gt;] lbug_with_loc+0x47/0xb0 [libcfs] [ 839.718362] [&lt;ffffffffa0c51911&gt;] __osd_oi_lookup+0x2f1/0x3e0 [osd_ldiskfs] [ 839.718654] [&lt;ffffffffa0c51aa6&gt;] osd_oi_lookup+0xa6/0x150 [osd_ldiskfs] [ 839.718941] [&lt;ffffffffa0c49d60&gt;] osd_object_init+0x5a0/0xcb0 [osd_ldiskfs] [ 839.719261] [&lt;ffffffffa05ad3ed&gt;] lu_object_alloc+0xcd/0x300 [obdclass] [ 839.719551] [&lt;ffffffffa05ad769&gt;] ? htable_lookup+0x119/0x1c0 [obdclass] [ 839.719817] [&lt;ffffffffa05adf55&gt;] lu_object_find_at+0x205/0x360 [obdclass] [ 839.720074] [&lt;ffffffff814fc8bc&gt;] ? __mutex_lock_slowpath+0x21c/0x2c0 [ 839.720342] [&lt;ffffffffa05b04ca&gt;] dt_locate_at+0x3a/0x140 [obdclass] [ 839.720602] [&lt;ffffffffa058ec2f&gt;] llog_osd_open+0x14f/0xbf0 [obdclass] [ 839.720895] [&lt;ffffffffa055a28d&gt;] llog_open+0xbd/0x2d0 [obdclass] [ 839.721206] [&lt;ffffffffa07501a2&gt;] llog_origin_handle_read_header+0x162/0x5e0 [ptlrpc] [ 839.721671] [&lt;ffffffffa0b7b064&gt;] mgs_handle+0xad4/0x11d0 [mgs] [ 839.721955] [&lt;ffffffffa0412361&gt;] ? libcfs_debug_msg+0x41/0x50 [libcfs] [ 839.722262] [&lt;ffffffffa0747898&gt;] ptlrpc_server_handle_request+0x3a8/0xc70 [ptlrpc] [ 839.722742] [&lt;ffffffffa04025ee&gt;] ? cfs_timer_arm+0xe/0x10 [libcfs] [ 839.723028] [&lt;ffffffffa0413e9f&gt;] ? lc_watchdog_touch+0x6f/0x170 [libcfs] [ 839.723349] [&lt;ffffffffa073efe1&gt;] ? ptlrpc_wait_event+0xb1/0x2a0 [ptlrpc] [ 839.723656] [&lt;ffffffff8105ad10&gt;] ? default_wake_function+0x0/0x20 [ 839.723961] [&lt;ffffffffa0748ba5&gt;] ptlrpc_main+0xa45/0x1650 [ptlrpc] [ 839.724266] [&lt;ffffffffa0748160&gt;] ? ptlrpc_main+0x0/0x1650 [ptlrpc] [ 839.724545] [&lt;ffffffff8100c10a&gt;] child_rip+0xa/0x20 [ 839.724845] [&lt;ffffffffa0748160&gt;] ? ptlrpc_main+0x0/0x1650 [ptlrpc] [ 839.725143] [&lt;ffffffffa0748160&gt;] ? ptlrpc_main+0x0/0x1650 [ptlrpc] [ 839.725426] [&lt;ffffffff8100c100&gt;] ? child_rip+0x0/0x20 [ 839.725683] [ 839.741236] Kernel panic - not syncing: LBUG </pre> </div></div> <p>I have a crashdump in /exports/crashdumps/192.168.10.221-2013-05-05-00\:56\:27<br/> source tag in my tree: master-20130505</p> LU-3278

osd_internal.h:747:osd_fid2oi()) ASSERTION( osd->od_oi_table != ((void *)0) && osd->od_oi_count >= 1 ) failed

Bug Major Resolved Duplicate nasf Oleg Drokin Sun, 5 May 2013 22:17:45 +0000 Thu, 7 Nov 2019 16:09:07 +0000 Tue, 7 May 2013 08:13:17 +0000 Lustre 2.4.0 0 5 <p>The direct reason for the failure is that the OI files may be not ready or already be closed when the LLOG processing. But the real issue is that the LLOG handling should not trigger OI files lookup.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">int</span> osd_oi_lookup(struct osd_thread_info *info, struct osd_device *osd, <span class="code-keyword">const</span> struct lu_fid *fid, struct osd_inode_id *id, bool check_fld) { <span class="code-keyword">if</span> (unlikely(fid_is_last_id(fid))) <span class="code-keyword">return</span> osd_obj_spec_lookup(info, osd, fid, id); ===&gt; <span class="code-keyword">if</span> ((check_fld &amp;&amp; fid_is_on_ost(info, osd, fid)) || fid_is_llog(fid)) <span class="code-keyword">return</span> osd_obj_map_lookup(info, osd, fid, id); <span class="code-keyword">if</span> (fid_is_fs_root(fid)) { osd_id_gen(id, osd_sb(osd)-&gt;s_root-&gt;d_inode-&gt;i_ino, osd_sb(osd)-&gt;s_root-&gt;d_inode-&gt;i_generation); <span class="code-keyword">return</span> 0; } <span class="code-keyword">if</span> (unlikely(fid_is_acct(fid))) <span class="code-keyword">return</span> osd_acct_obj_lookup(info, osd, fid, id); <span class="code-keyword">if</span> (!osd-&gt;od_igif_inoi &amp;&amp; fid_is_igif(fid)) { osd_id_gen(id, lu_igif_ino(fid), lu_igif_gen(fid)); <span class="code-keyword">return</span> 0; } <span class="code-keyword">return</span> __osd_oi_lookup(info, osd, fid, id); } </pre> </div></div> <p>That means the "fid_is_llog(fid)" returned false for this case.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> inline <span class="code-object">int</span> fid_is_llog(<span class="code-keyword">const</span> struct lu_fid *fid) { <span class="code-comment">/* file with OID == 1 is not llog but contains last oid */</span> <span class="code-keyword">return</span> fid_seq_is_llog(fid_seq(fid)) &amp;&amp; fid_oid(fid) &gt; 1; } </pre> </div></div> <p>But the old "fid_is_llog(fid)" was like that:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> inline <span class="code-object">int</span> fid_is_llog(<span class="code-keyword">const</span> struct lu_fid *fid) { <span class="code-keyword">return</span> fid_seq_is_llog(fid_seq(fid)); } </pre> </div></div> <p>I am not sure whether it is related with above code changes. I have add debug information in the patch (<a href="http://review.whamcloud.com/#change,6267" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6267</a>) to catch the "@fid" when the failure reproduced.</p> <p>The file with OID == 1 is not llog but /seq-1-lastid which stores last generated fid oid for llogs, it doesn't need osd_obj_map_xxx, it is just ordinary file in the root directory. But you said above it can be used before OI is initialized, probably that is the reason, though I am not sure why llog is allowed to work over OSD which is not ready, probably OI should be set up early?</p> <p>More study the log and code analysis, I think the failure caused by that the OI files closed before the MGS service thread stopped. So the solution is NOT setup OI early, instead, we should cleanup the OI some later.</p> <p>It is another instance of <a href="https://jira.whamcloud.com/browse/LU-3275" title="sanity-scrub.sh test_11: osd_seq_load() ASSERTION( map-&gt;om_root ) failed" class="issue-link" data-issue-key="LU-3275"><del>LU-3275</del></a>. I have made patch for such bug.</p> <p><a href="http://review.whamcloud.com/#change,6267" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,6267</a></p> <p>Another failure instance of <a href="https://jira.whamcloud.com/browse/LU-3275" title="sanity-scrub.sh test_11: osd_seq_load() ASSERTION( map-&gt;om_root ) failed" class="issue-link" data-issue-key="LU-3275"><del>LU-3275</del></a></p> <p>not sure whether to file a new ticket, but I've hit this locally few times yet:</p> <p>Lustre: DEBUG MARKER: == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ====================== 20:38:18 (1573141098)<br/> LustreError: 30344:0:(osd_internal.h:994:osd_fid2oi()) ASSERTION( osd-&gt;od_oi_table != NULL &amp;&amp; osd-&gt;od_oi_count &gt;= 1 ) failed: <span class="error">&#91;0xa:0x9:0x0&#93;</span><br/> LustreError: 30344:0:(osd_internal.h:994:osd_fid2oi()) LBUG<br/> Call Trace:<br/> ? __schedule+0x2ad/0xb00<br/> schedule+0x34/0x80<br/> lbug_with_loc+0x79/0x80 <span class="error">&#91;libcfs&#93;</span><br/> ? __osd_oi_lookup.isra.0+0x32b/0x3d0 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> ? osd_object_init+0x290/0x2110 <span class="error">&#91;osd_ldiskfs&#93;</span><br/> ? __raw_spin_lock_init+0x28/0x50<br/> ? lu_object_start.isra.0+0x68/0x100 <span class="error">&#91;obdclass&#93;</span><br/> ? lu_object_find_at+0x30b/0x9e0 <span class="error">&#91;obdclass&#93;</span><br/> ? dt_locate_at+0x13/0xa0 <span class="error">&#91;obdclass&#93;</span><br/> ? llog_osd_open+0x2b0/0xf90 <span class="error">&#91;obdclass&#93;</span><br/> ? __raw_spin_lock_init+0x28/0x50<br/> ? llog_open+0x1a7/0x3f0 <span class="error">&#91;obdclass&#93;</span><br/> ? llog_origin_handle_read_header+0x178/0x500 <span class="error">&#91;ptlrpc&#93;</span><br/> ? tgt_llog_read_header+0x26/0xb0 <span class="error">&#91;ptlrpc&#93;</span><br/> ? tgt_request_handle+0x411/0x17f0 <span class="error">&#91;ptlrpc&#93;</span></p> Duplicate LU-3275 Development Rank 1|hzvq6n: Rank (Obsolete) 8115 Severity