https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7010 Lustre <p>Recent recovery issues in Maloo show the following:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">00000040:00020000:1.0:1439669743.543046:0:5740:0:(llog.c:489:llog_process_thread()) Local llog found corrupted 00000040:00100000:1.0:1439669743.545890:0:5740:0:(llog.c:167:llog_cancel_rec()) Canceling 1 in log 0x1:1024 00000040:00100000:1.0:1439669743.546205:0:5740:0:(llog.c:167:llog_cancel_rec()) Canceling 64838 in log 0x1:1024 00000040:00100000:1.0:1439669743.546229:0:5740:0:(llog.c:167:llog_cancel_rec()) Canceling 64864 in log 0x1:1024 00000040:00100000:1.0:1439669743.546242:0:5740:0:(llog.c:167:llog_cancel_rec()) Canceling 64896 in log 0x1:1024 00000040:00100000:1.0:1439669743.546254:0:5740:0:(llog.c:167:llog_cancel_rec()) Canceling 64897 in log 0x1:1024 00000040:00100000:1.0:1439669743.546267:0:5740:0:(llog.c:167:llog_cancel_rec()) Canceling 64899 in log 0x1:1024 </pre> </div></div> <p>As I can see, the DNE2 'update recovery' may return -EIO error if some update was applied with error. That cause whole llog processing to stop and cancel all other updates. After that recovery stops with various errors.</p> <p>Here is an example, test_70b:<br/> <a href="https://testing.hpdd.intel.com/test_sets/1a8282a6-43d8-11e5-a4bc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1a8282a6-43d8-11e5-a4bc-5254006e85c2</a></p> LU-7010

"Local llog found corrupted" during DNE2 recovery

Bug Critical Resolved Duplicate Di Wang Mikhail Pershin Sun, 16 Aug 2015 07:38:19 +0000 Mon, 21 Sep 2015 05:25:05 +0000 Mon, 21 Sep 2015 05:25:05 +0000 Lustre 2.8.0 0 4 <p>Di is taking care of this one</p> <p>Hmm, It seems to me, this corruption is related this unlanded patch <a href="http://review.whamcloud.com/#/c/14912/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14912/</a> And this patch did change something inside llog, so it is quite possible related with this specific patch.<br/> Mike, did you see this failure happen on other patches? . If not, I will close this ticket.</p> <p>I see that this problem is not related with that patch, it already exists in code. See, the llog_process_thread() has an old code about the corruption handling:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (unlikely(rc == -EIO &amp;&amp; loghandle-&gt;lgh_obj != NULL)) { /* something bad happened to the processing of a local * llog file, probably I/O error or the log got corrupted.. * to be able to <span class="code-keyword">finally</span> release the log we discard any * remaining bits in the header */ CERROR(<span class="code-quote">"Local llog found corrupted\n"</span>); <span class="code-keyword">while</span> (index &lt;= last_index) { <span class="code-keyword">if</span> (ext2_test_bit(index, LLOG_HDR_BITMAP(llh)) != 0) llog_cancel_rec(lpi-&gt;lpi_env, loghandle, index); index++; } rc = 0; } </pre> </div></div> <p>Meanwhile the update_recovery.c introduced new callbacks which may return -EIO. Technically that doesn't mean the llog itself is corrupted, and we shouldn't cancel all other llog records. I think we shouldn't use EIO error code there at all.</p> <p>Mike: I understand the corrupt checking code is already there. But I mean the reason to cause this corruption is quite related with the change, since this "Local llog found corruption" error seems happen on every run of that patch. I did not see this on other patch. Do I miss sth? Do you know why did it return -EIO? could you please explain here. Thanks.</p> <p>It is not only happening with that patch, in fact I don't see even how it can be related. Meanwhile it looks like duplicate of <a href="https://jira.whamcloud.com/browse/LU-6844" title="replay-single test 70b failure: &#39;rundbench load on * failed!&#39;" class="issue-link" data-issue-key="LU-6844"><del>LU-6844</del></a>, some reports there also have the same problem with corrupted log. E.g. check report <a href="https://testing.hpdd.intel.com/test_sets/1a8282a6-43d8-11e5-a4bc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1a8282a6-43d8-11e5-a4bc-5254006e85c2</a>, MDS1 console log contains:<br/> 20:15:58:LustreError: 5740:0:(llog.c:489:llog_process_thread()) Local llog found corrupted</p> <p>I am not sure about the reason, but I see that -EIO from llog callbacks also will cause this message and llog cancelling, maybe update llog processing callback may return -EIO?</p> <p>IMHO, this EIO usually comes from llog_osd_next_block(), which means the local llog is indeed corrupted. Hmm, <a href="https://testing.hpdd.intel.com/test_sets/1a8282a6-43d8-11e5-a4bc-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/1a8282a6-43d8-11e5-a4bc-5254006e85c2</a> is also from the patch 14912, do I miss sth?<br/> Besides, there is an obvious mistake in 14912, which might corrupt the update llog.</p> <p>Hmm, I checked the failure in <a href="https://jira.whamcloud.com/browse/LU-6844" title="replay-single test 70b failure: &#39;rundbench load on * failed!&#39;" class="issue-link" data-issue-key="LU-6844"><del>LU-6844</del></a>, I do not think they are related. Most failures there are either due to "No space left". </p> <p>what mistake in 14912 do you mean, could you explain, about llh_size? If some reports from <a href="https://jira.whamcloud.com/browse/LU-6844" title="replay-single test 70b failure: &#39;rundbench load on * failed!&#39;" class="issue-link" data-issue-key="LU-6844"><del>LU-6844</del></a> are also related to patch 14912 then it can be the reason, I agree. Let's wait then for updated patch first.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>what mistake in 14912 do you mean, could you explain, about llh_size? </pre> </div></div> <p>Yes, it can not use llh_size to calculate the write offset, because for update log, even it is NOT fixed size update record, llh_size are still &gt; 0. So if the write_offset is wrong, then new write will ruin the llog anyway. </p> <p>this is clearly caused by patch <a href="http://review.whamcloud.com/#/c/14912/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/14912/</a> , since that patch is not landed yet. I will close this one to avoid duplicate efforts.</p> <p>duplicate with <a href="https://jira.whamcloud.com/browse/LU-6556" title="changelog catalog corruption if all possible records is define " class="issue-link" data-issue-key="LU-6556"><del>LU-6556</del></a></p> Related LU-7011 LU-5716 Development Rank 1|hzxklb: Rank (Obsolete) 9223372036854775807 Severity