https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13061 Lustre <p>If a system has hit <a href="https://jira.whamcloud.com/browse/LU-12593" title="update_log corruption" class="issue-link" data-issue-key="LU-12593"><del>LU-12593</del></a> with a corrupt block in the llog file, it may trigger an <tt>LASSERT()</tt> because of a bad FID found in the unintialized part of the block. Applying the patch from that ticket is too late to fix the problem.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LustreError: 17438:0:(osd_handler.c:1077:osd_fid_lookup()) ASSERTION( fid_is_sane(fid) || fid_is_idif(fid) ) failed: [0x0:0x68:0x0] LustreError: 17438:0:(osd_handler.c:1077:osd_fid_lookup()) LBUG Pid: 17438, comm: llog_process_th 3.10.0-1062.1.1.el7_lustre.x86_64 Call Trace: libcfs_call_trace+0x8c/0xc0 [libcfs] lbug_with_loc+0x4c/0xa0 [libcfs] osd_fid_lookup+0xc8/0x1c60 [osd_ldiskfs] osd_object_init+0x61/0x110 [osd_ldiskfs] lu_object_start.isra.35+0x8b/0x120 [obdclass] lu_object_find_at+0x1e1/0xa60 [obdclass] dt_locate_at+0x1d/0xb0 [obdclass] llog_osd_open+0x50e/0xf30 [obdclass] llog_open+0x15a/0x3e0 [obdclass] osp_sync_init+0x44a/0xe20 [osp] osp_init0.isra.19+0x1aed/0x1f60 [osp] osp_device_alloc+0x86/0x130 [osp] obd_setup+0x119/0x280 [obdclass] class_setup+0x2a8/0x840 [obdclass] class_process_config+0x1726/0x2830 [obdclass] class_config_llog_handler+0x819/0x1520 [obdclass] llog_process_thread+0x82f/0x18e0 [obdclass] llog_process_thread_daemonize+0x9f/0xe0 [obdclass] </pre> </div></div> <p>Lustre should avoid ever triggering an LASSERT() on data read from disk or from the network. In this case, it probably makes sense to add a check in <tt>llog_osd_open()</tt> with <tt>fid_is_sane()</tt> before it uses the FID, and just return an error rather than crashing.</p> LU-13061

osd_fid_lookup()) ASSERTION( fid_is_sane(fid) || fid_is_idif(fid) ) failed: [0x0:0x68:0x0]

Bug Critical Resolved Fixed Hongchao Zhang Andreas Dilger Tue, 10 Dec 2019 23:40:35 +0000 Wed, 22 Jan 2020 12:56:14 +0000 Fri, 10 Jan 2020 13:07:03 +0000 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.12.4 0 4 <p>HongChao, can you please make a patch for this, preferably with a test case. You can add a <tt>CFS_FAIL_LOC()</tt> check to set <tt>fid_seq=0</tt> or similar to trigger the <tt>LASSERT()</tt>, then verify your patch fixes it.</p> <p>Based on further analysis of the problem, it seems we can't verify <tt>fid_is_sane()</tt> directly in <tt>osp_sync_llog_init()</tt> since the logid is not yet converted to a FID. If <tt>llog_osd_open()</tt> checks <tt>fid_is_sane()</tt> and returns <tt>-ENOENT</tt> if it isn't sane:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (logid != NULL) { logid_to_fid(logid, &amp;lgi-&gt;lgi_fid); <span class="code-keyword">if</span> (!fid_is_sane(&amp;lgi-&gt;lgi_fid)) { CERROR(<span class="code-quote">"%s: bad FID "</span>DFID<span class="code-quote">" in %s idx: rc = %d\n"</span>, ctxt-&gt;loc_exp-&gt;exp_obd-&gt;obd_name, PFID(&amp;lgi-&gt;lgi_fid), -ENOENT); RETURN(-ENOENT); } <span class="code-keyword">else</span> { /* If logid == NULL, then it means the caller needs </pre> </div></div> <p>then this return code is checked in <tt>osp_sync_llog_init()</tt> and the llog will be recreated:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> rc = llog_open(env, ctxt, &amp;lgh, &amp;osi-&gt;osi_cid.lci_logid, NULL, LLOG_OPEN_EXISTS); <span class="code-comment">/* re-create llog <span class="code-keyword">if</span> it is missing */</span> <span class="code-keyword">if</span> (rc == -ENOENT) logid_set_id(&amp;osi-&gt;osi_cid.lci_logid, 0); </pre> </div></div> <p>Hi Andreas, <br/> How about calling 'logid_to_fid" in osp_sync_llog_init after getting the logid from "CATALOGS" file, then check whether it is sane?</p> <p>Sure, that would also work, and is also much more direct.</p> <p>There is a bit of a question in my mind whether <tt>logid</tt> may <em>not</em> be a FID in some cases (e.g. upgrade from some very old system), but I don't <em>think</em> that is true anymore. This compatibility was needed for old filesystems when they were upgraded from pre-2.4 <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Lustre, but all of these old systems should be gone at this point, or are never upgrading.</p> <p>After fixing the current LASSERT issue, it might make sense to look at the use of <tt>llog_logid</tt> in the code and update it to use <tt>lu_fid</tt> wherever possible, by converting the <tt>llog_logid</tt> into <tt>lu_fid</tt> immediately and only passing the FID around. The next step would be to add a new <tt>LLOG_LOGFID_MAGIC</tt> that only stores the <tt>lu_fid</tt> to the catalog, start using it after some delay (with <tt>OBD_VERSION_CHECK()</tt> and/or backport code to LTS to read <tt>LLOG_LOGFID_MAGIC</tt> records and convert into <tt>llog_logid</tt> to minimize code change but still allow upgrade/downgrade to work) and then eventually we can remove <tt>llog_logid</tt>.</p> <p>as per the dump of the "CATALOGS", the content of the "llog_catid" is</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>000020 0000000400000068 0000000000000000 000030 0000000000000000 0000000000000000 </pre> </div></div> <p>the OID field is "0x400000068", but is truncated to "0x68" during assigning it to lu_fid.f_oid(32bits).<br/> the data seems to be a normal FID (f_seq=0x400000068, f_oid=0, f_ver=0)</p> <blockquote> <p>the OID field is "0x400000068", but is truncated to "0x68" during assigning it to lu_fid.f_oid(32bits).<br/> the data seems to be a normal FID (f_seq=0x400000068, f_oid=0, f_ver=0)</p></blockquote> <p>No, the <tt>0x400000068</tt> field is the <tt>oi_id</tt> field (mapped to <tt>f_oid</tt>), and the <tt>f_seq</tt> field is <tt>0x0</tt>, which is what triggers the LASSERT. Valid records look like the following with <tt>oi_seq = f_seq = 0x1 = FID_SEQ_LLOG</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>000080 00000000000004ef 0000000000000001 000090 0000000000000000 0000000000000000 </pre> </div></div> <p>In this case, the FID is <tt><span class="error">&#91;0x1:0x4ef:0x0&#93;</span></tt> and would map to object <tt>O/1/d15/1263</tt> on the MDT.</p> <p>Hongchao Zhang (hongchao@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36998" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36998</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13061" title="osd_fid_lookup()) ASSERTION( fid_is_sane(fid) || fid_is_idif(fid) ) failed: [0x0:0x68:0x0]" class="issue-link" data-issue-key="LU-13061"><del>LU-13061</del></a> osp: check catlog FID after reading in<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7a5df8c9c0fa2cab2135e68e305a9a850263d720</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36998/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36998/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13061" title="osd_fid_lookup()) ASSERTION( fid_is_sane(fid) || fid_is_idif(fid) ) failed: [0x0:0x68:0x0]" class="issue-link" data-issue-key="LU-13061"><del>LU-13061</del></a> osp: check catlog FID after reading in<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4597fa7d884de0f1a1b030052d4d34983fed6109</p> <p>Landed for 2.14</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37185" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37185</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13061" title="osd_fid_lookup()) ASSERTION( fid_is_sane(fid) || fid_is_idif(fid) ) failed: [0x0:0x68:0x0]" class="issue-link" data-issue-key="LU-13061"><del>LU-13061</del></a> osp: check catlog FID after reading in<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 0d2e23757ef6663f8b06b935253835937d8ca1f3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37185/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37185/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13061" title="osd_fid_lookup()) ASSERTION( fid_is_sane(fid) || fid_is_idif(fid) ) failed: [0x0:0x68:0x0]" class="issue-link" data-issue-key="LU-13061"><del>LU-13061</del></a> osp: check catlog FID after reading in<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 055eab6bd4c29bc961a10824ffa44323cce7640c</p> Duplicate Related LU-5369 LU-12593 Development Rank 1|i00qr3: Rank (Obsolete) 9223372036854775807 Severity