https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7011 Lustre <p>While working on <a href="https://jira.whamcloud.com/browse/LU-6696" title="ASSERTION( rc == 0 || rc == LLOG_PROC_BREAK ) failed: 0 changes, 0 in progress, 0 in flight: -5" class="issue-link" data-issue-key="LU-6696"><del>LU-6696</del></a> ticket the tool to repair corrupted llog catalogs were introduced. The same job could be done in kernel code to repair llogs online if possible.</p> LU-7011

Kernel part of llog subsystem can do self-repairing in some cases

Improvement Major Open Unresolved Mikhail Pershin Mikhail Pershin Sun, 16 Aug 2015 11:34:09 +0000 Tue, 20 Oct 2020 14:32:58 +0000 Lustre 2.8.0 0 8 <p>I think we need both the tool and online repair. Start with tool for now.</p> <p>Tool is <a href="http://review.whamcloud.com/15245" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15245</a></p> <p>It makes more sense to have the llog code repair, skip, and/or clear broken records rather than using an external tool. If the external tool can detect and fix these problems (after the user's MDS has crashed and waited all night for them to figure out the problem and run the tool), why not just add enough checks into the llog processing to clean it up immediately? That avoids the MDS downtime, and avoids the need for the user to even know an llog repair tool exists and that they need to run it.</p> <p>Mike, just curious, will you check/repair both catalog and plain log in this patch? Thanks </p> <p>Andreas, I agree in general, the difference is that we are more restricted inside kernel, e.g. we can't just do repair in the current context, but have to start separate repair thread, exclusively accessing that llog. I mean that tool is much simpler to implement than auto-repair, there are no problems with concurrent access, transactions, etc. Meanwhile I agree that auto-repair in preferable and I am going to implement some basic checks/repairs at least.</p> <p>Di, it is possible only for fixed size llog in fact, the catalog is the only real example we have now.</p> <p>It seems possible to do at least some basic repair of variable-sized llog records. For example, if a corrupt llog record is found (i.e. hdr len != tail len), one option would be to scan the rest of the chunk for potential matching llog hdr/tail pairs that allow resyncing the stream. A second option (easier to implement, but recovers fewer logs) would be to jump to the start of the next llog chunk and clear the records between the corrupt chunk and the start of the new chunk.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Di, it is possible only for fixed size llog in fact, the catalog is the only real example we have now. </pre> </div></div> <p>Yes, it seems except catalog, all important plain logs are not fixed size. change log, config log, update log, unlink log are all not fixed size. Unfortunately, most of the corruptions seems happen in plain log, at least that happens in DNE test. Hmm, actually most of them are header and tail are not matched to each other, (lrh_len != tail_len or lrh_idx != tail_index), Hmm, some of them even use LASSERT to check, probably we should change that to CERROR.</p> <p>In fact I think we should find the reason of these issues with bad tail, it is not normal behavior and something is wrong there definitely. I mean it is not some sort of corruption due to disk issues, etc., but issue in our code causing that.</p> Related LU-6696 LU-7010 Development Rank 1|hzxklj: Rank (Obsolete) 9223372036854775807