https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15938 Lustre <p>A broken DNE recovery llog record was preventing MDT-MDT recovery from completing. MDT0003 was permanently unable to finish recovery with MDT0019, looping on:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>llog_process_thread()) lfs02-MDT0019-osp-MDT0003 retry remote llog process </pre> </div></div> <p>There was a bad record in the llog file, and the recovery would process the llog (all but one other record had successfully been cancelled) and then hit a bad record and abort, then retry.</p> <p>Since the DNE recovery llog for MDT0003 is stored on MDT0019, this necessitated "fixing" the llog file on MDT0019 by truncating it to zero bytes and which allowed MDT0003 recovery to finish.</p> <p>Retrying recovery can be useful in some cases, if the remote MDT is inaccessible, but if there is a single bad record it makes sense to only retry once (in case the llog was in the middle of being written) and then cancel this record and continue with the rest of recovery, or at worst abort recovery with that MDT and cancel the whole llog file. Otherwise, this needs manual intervention to recover from this situation, which can't do better than cancelling the llog record (pending <a href="https://jira.whamcloud.com/browse/LU-15937" title="lctl llog commands do not work for DNE recovery logs" class="issue-link" data-issue-key="LU-15937">LU-15937</a>) or delete the whole llog file.</p> LU-15938

MDT recovery did not finish due to corrupt llog record

Bug Minor Resolved Fixed Mikhail Pershin Andreas Dilger Sun, 12 Jun 2022 03:24:19 +0000 Mon, 5 Jun 2023 16:11:02 +0000 Thu, 8 Dec 2022 00:14:59 +0000 Lustre 2.14.0 Lustre 2.16.0 0 14 <p>This is similar to <a href="https://jira.whamcloud.com/browse/LU-15761" title="cannot finish MDS recovery" class="issue-link" data-issue-key="LU-15761"><del>LU-15761</del></a> and <a href="https://jira.whamcloud.com/browse/LU-15645" title="gap in recovery llog should not be a fatal error" class="issue-link" data-issue-key="LU-15645"><del>LU-15645</del></a>, but apparently the llog error is different enough that those patches did not allow the recovery to complete automatically.</p> <p>where can I find that corrupted llog for analysis?</p> <p>attached the requested log</p> <p>Mike</p> <p>Could you please investigate</p> <p>Thanks</p> <p>Peter</p> <p>The problematic llog file decoded by <tt>llog_reader</tt> from master shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rec #7707 type=106a0000 len=1160 offset 8670472 Header size : 32768 llh_size : 496 Time : Thu Apr 7 10:52:40 2022 Number of records: 2 cat_idx: 9 last_idx: 8244 Target uuid : ----------------------- #7707 (1160)updatelog record master_transno:45507493823 batchid:37129103921 flags:0x0 u_index:0 u_count:11 p_count:18 [0x1840003b3b:0x2fce:0x0] type:create/1 params:2 p_0:0 p_1:1 [0x1840003b3b:0x2fce:0x0] type:ref_add/3 params:0 [0x1840003b3b:0x2fce:0x0] type:insert/10 params:3 p_0:2 p_1:3 p_2:4 [0x1840003b3b:0x2fce:0x0] type:insert/10 params:3 p_0:5 p_1:1 p_2:4 [0x1840003b3b:0x2fce:0x0] type:xattr_set/7 params:3 p_0:6 p_1:7 p_2:8 [0x1280002b0e:0x1be36:0x0] type:insert/10 params:3 p_0:9 p_1:3 p_2:10 [0x1280002b0e:0x1be36:0x0] type:ref_add/3 params:0 [0x1840003b3b:0x2fce:0x0] type:xattr_set/7 params:3 p_0:11 p_1:12 p_2:8 [0x1280002b0e:0x1be36:0x0] type:attr_set/5 params:1 p_0:13 [0x1840003b3b:0x2fce:0x0] type:xattr_set/7 params:3 p_0:14 p_1:15 p_2:8 [0x200000001:0x15:0x0] type:write/12 params:2 p_0:16 p_1:17 p_0 - 208/\x8E070000000000000000000000000000000000000000000000000000000000000000000000000000005ob\x00000000005ob\x00000000005ob\x0000 p_1 - 16/\x0E+\x0080120000006\xBE0100000000 p_2 - 2/. p_3 - 0/ p_4 - 0/ p_5 - 16384/\x0000000000000400jb\x0000000000@\x0000000000000300jb\x00000000..\x0000000000000C00000000000000trusted.dmv\x00d\x0000000\x00 p_6 - 0/ p_7 - 0/ p_8 - 0/ p_9 - 0/ p_10 - 0/ p_11 - 0/ p_12 - 0/ p_13 - 0/ p_14 - 0/ p_15 - 0/ p_16 - 0/ p_17 - 0/ #8244 (1160)NOT SET updatelog record master_transno:45508195003 batchid:37129117505 flags:0x0 u_index:0 u_count:11 p_count:18 [0x18400034de:0x6863:0x0] type:create/1 params:2 p_0:0 p_1:1 [0x18400034de:0x6863:0x0] type:ref_add/3 params:0 [0x18400034de:0x6863:0x0] type:insert/10 params:3 p_0:2 p_1:3 p_2:4 [0x18400034de:0x6863:0x0] type:insert/10 params:3 p_0:5 p_1:1 p_2:4 [0x18400034de:0x6863:0x0] type:xattr_set/7 params:3 p_0:6 p_1:7 p_2:8 [0x128000369b:0x174b8:0x0] type:insert/10 params:3 p_0:9 p_1:3 p_2:10 [0x128000369b:0x174b8:0x0] type:ref_add/3 params:0 [0x18400034de:0x6863:0x0] type:xattr_set/7 params:3 p_0:11 p_1:12 p_2:8 [0x128000369b:0x174b8:0x0] type:attr_set/5 params:1 p_0:13 [0x18400034de:0x6863:0x0] type:xattr_set/7 params:3 p_0:14 p_1:15 p_2:8 [0x200000001:0x15:0x0] type:write/12 params:2 p_0:16 p_1:17 p_0 - 208/\x8E070000000000000000000000000000000000000000000000000000000000000000000000000000H5ob\x00000000H5ob\x00000000H5ob\x000000 p_1 - 16/\x9B6\x008012000000B8t\x0100000000 p_2 - 2/. </pre> </div></div> <p>The recovery from the source MDT reported an error when processing record 8245.</p> <p>yes, I've found the same. Log itself is not corrupted, all records are processed by llog_reader without errors, llog size is also fits data in llog. What is not OK so far - "<tt>Number of records: 2</tt>", first I thought that is OK because one record is llog_header but in fact llog_reader handles that and does decrement. So that is mismatch, llog reports 2 live records but has only 1. I am not sure if that is problem or not. </p> <p>I am checking code to see how <tt>-5</tt> could be returned. My idea is that we might have read over llog size somehow</p> <p>Other observation is that used server version misses the following patches:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> LU-13974 llog: check stale osp object LU-15645 obdclass: llog to handle gaps LU-15761 obdclass: fix locking in llog_cat_refresh() </pre> </div></div> <p> It looks we don't have here direct result of any of them but still it would be better to update</p> <p>just addition to comment above, lgh_count mismatch may be caused by undo operations over llog, if pad record was involved, so that might indicate that there were undo operations over llog</p> <p>From the debug logs it shows the MDT is getting -EIO when trying to read the next log, maybe because the llog is short, or it <em>thinks</em> it is short because of the number of records?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000001:1.0:1654980511.161770:0:11204:0:(osp_md_object.c:1345:osp_md_read()) Process leaving (rc=4400 : 4400 : 1130) 00000040:00000001:1.0:1654980511.161772:0:11204:0:(llog_osd.c:1089:llog_osd_next_block()) Process leaving via out (rc=18446744073709551611 : -5 : 0xfffffffffffffffb) 00000040:00000001:1.0:1654980511.161774:0:11204:0:(lustre_log.h:465:llog_next_block()) Process leaving (rc=18446744073709551611 : -5 : fffffffffffffffb) 00000040:00000001:1.0:1654980511.161775:0:11204:0:(llog.c:572:llog_process_thread()) Process leaving via out (rc=18446744073709551611 : -5 : 0xfffffffffffffffb) 00000040:00080000:1.0:1654980511.161777:0:11204:0:(llog.c:761:llog_process_thread()) stop processing plain 0x2:2147506325:0 index 8245 count 3 00000040:00020000:1.0:1654980511.161778:0:11204:0:(llog.c:774:llog_process_thread()) lfs02-MDT0019-osp-MDT0003 retry remote llog process 00000040:00000001:1.0:1654980511.161781:0:11204:0:(llog.c:897:llog_process_or_fork()) Process leaving (rc=18446744073709551605 : -11 : fffffffffffffff5) 00000040:00000001:1.0:1654980511.161782:0:11204:0:(llog_cat.c:904:llog_cat_process_cb()) Process leaving (rc=18446744073709551605 : -11 : fffffffffffffff5) 00000040:00000001:1.0:1654980511.161784:0:11204:0:(llog.c:742:llog_process_thread()) Process leaving via out (rc=18446744073709551605 : -11 : 0xfffffffffffffff5) 00000040:00080000:1.0:1654980511.161785:0:11204:0:(llog.c:761:llog_process_thread()) stop processing catalog 0x1:2147484688:0 index 9 count 12 00000040:00000001:1.0:1654980511.161790:0:11204:0:(llog.c:897:llog_process_or_fork()) Process leaving (rc=18446744073709551605 : -11 : fffffffffffffff5) 00000040:00000001:1.0:1654980511.161791:0:11204:0:(llog_cat.c:966:llog_cat_process_or_fork()) Process leaving (rc=18446744073709551605 : -11 : fffffffffffffff5) </pre> </div></div> <p>so far it looks like MDT0003 has record #8245 in local llog bitmap and is trying to read that records from MDT0019. Meanwhile MDT0019 doesn't have it neither in bitmap nor in file, file size ends exactly at #8244 record. That means that #8245 write request were sent to MDT0019 from the MDT0003, that is why it has it in bitmap but wasn't yet written on MDT0019 due to some reason but concurrent llog processing reading it.</p> <p>As for loop in code, llog_osd_next_block() reads llog up to its end, get last_idx in llog as #8244, since it is less than expected #8245, it does read again, but cycle exists because current offset is equal to llog_size at that moment causing -5 error. I am not yet sure what happens next, if caller would try read again and again then needed update should arrive at some moment and read would succeeded but we don't see that. So either write is just lost completely and never happens or caller doesn't retry that read properly. </p> <p><a href="https://review.whamcloud.com/47003" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47003</a> - could solve read arriving prior related write<br/> It is still unclear why/could write be lost at all, so read will retry loop forever</p> <p>So, at the moment the first problem to address is to prevent such llog processing loop if needed update is lost (presumably), at least we can't wait for it forever, probably we could check if there is related write inflight or so.</p> <p>Second problem to address is <tt>abort_recovery_mdt</tt> option, being  used it skips update recovery but doesn't remove related update logs, so if we have llog corruption situation which need to be solved, the <tt>abort_recovery_mdt</tt> doesn't resolve it, every server remount would stuck on the same issues. I think if we decided to abort/skip update recovery there is no sense to keep update logs and they should be truncated. This is to be discussed, probably I am missing some cases when they must stay</p> <p> </p> <p>speaking of <tt>abort_recovery_mdt</tt> in the <tt>lod_sub_recovery_thread()</tt> the cycle breaks only on <tt>abort_recovery</tt> being set, so <tt>abort_recovery_mdt</tt> doesn't make things better but also doesn't stop that thread. That could explain why setting <tt>abort_recovery_mdt</tt> doesn't help with stopping stuck MDT-MDT recovery</p> <p>Ok, so two problems then. The llog is permanently short for whatever reason (posdibly old non-atomic writes?), so retrying forever will never fix it. Having a few retries might be OK in case the write is in progress. Maybe we need a separate error code for a short read?</p> <p>Secondly, abort_revov_mdt should definitely abort the MDT recovery. It also makes sense that this will clear or reset or delete the recovery logs so that they don't try to be re-used on the next recovery. If recovery is already aborted and the filesystem is allowed to be used, then a later replay of old recovery logs may actually be bad. It also means that any problems in the recovery logs that are not "self healing" will be hit on every restart. </p> <p>yes, the same thoughts here. I want to figure out first why write was lost, if it wasn't committed, I'd expect it to be replayed anyway. So probably retry is done not correctly, e.g. record id is wrong or something else, or that could be result of <tt>abort_recovery_mdt</tt> attempt, which skips update recovery, i.e. updates are not re-applied and then it is stuck in retry for missing writes.</p> <p>Anyway, despite the reason I think retry mechanism should presume that write might be lost and stops at some moment. I think that is covered by Alex patch <a href="https://review.whamcloud.com/#/c/47003/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/47003/</a> already, need to discuss with him. For me it looks like with that patch there might be no need in retry at all, at least if we are sure that read error is returned from another server but is not network error.</p> <p>"Mike Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47698" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47698</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> lod: prevent endless retry in recovery thread<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2c0135579c350a897313b4ff74a1cc7aea200ec7</p> <p>Initial approach which resolves endless retry loop we've observed due to remote llog short read handling. Patch also makes obd_abort_recovery_mdt option to abort update recovery threads as well. That should prevent known endless retry cases and make possible manual intervention by using abort_recovery_mdt parameter if update recovery would stuck due to network problems.</p> <p>As noted above, more work is needed to remove update llogs upon abort_recovery_mdt setting and it is worth to think about limit for number of retries when remote server is not accessible, so far I have no idea what to choose as basis - recovery hard (or soft) timeout value maybe?</p> <p>OK, with more analysis, I check the <tt>llog_reader</tt> and find out it doesn't report all inconsistencies. Modified version showed problem in update_log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # llog_reader update_log  rec #7707 type=106a0000 len=1160 offset 8670472 in bitmap: rec #8245 is set! llog has 1 records but header count is 2 Header size : 32768     llh_size : 496 Time : Thu Apr  7 12:52:40 2022 <span class="code-object">Number</span> of records: 2    cat_idx: 9    last_idx: 8244 Target uuid :  ----------------------- #7707 (1160) id: 0 updatelog record master_transno:45507493823 batchid:37129103921 flags:0x0 u_index:0 u_count:11 p_count:18     [0x1840003b3b:0x2fce:0x0] type:create/1 params:2 p_0:0 p_1:1      [0x1840003b3b:0x2fce:0x0] type:ref_add/3 params:0      [0x1840003b3b:0x2fce:0x0] type:insert/10 params:3 p_0:2 p_1:3 p_2:4      [0x1840003b3b:0x2fce:0x0] type:insert/10 params:3 p_0:5 p_1:1 p_2:4      [0x1840003b3b:0x2fce:0x0] type:xattr_set/7 params:3 p_0:6 p_1:7 p_2:8      [0x1280002b0e:0x1be36:0x0] type:insert/10 params:3 p_0:9 p_1:3 p_2:10      [0x1280002b0e:0x1be36:0x0] type:ref_add/3 params:0      [0x1840003b3b:0x2fce:0x0] type:xattr_set/7 params:3 p_0:11 p_1:12 p_2:8      [0x1280002b0e:0x1be36:0x0] type:attr_set/5 params:1 p_0:13      [0x1840003b3b:0x2fce:0x0] type:xattr_set/7 params:3 p_0:14 p_1:15 p_2:8      [0x200000001:0x15:0x0] type:write/12 params:2 p_0:16 p_1:17      p_0 - 208/\x8E070000000000000000000000000000000000000000000000000000000000000000000000000000005ob\x00000000005ob\x00000000005ob\x0000     p_1 - 16/\x0E+\x0080120000006\xBE0100000000     p_2 - 2/.     p_3 - 0/     p_4 - 0/     p_5 - 16384/\x0000000000000400jb\x0000000000@\x0000000000000300jb\x00000000..\x0000000000000C00000000000000trusted.dmv\x00d\x0000000\x00     p_6 - 0/     p_7 - 0/     p_8 - 0/     p_9 - 0/     p_10 - 0/     p_11 - 0/     p_12 - 0/     p_13 - 0/     p_14 - 0/     p_15 - 0/     p_16 - 0/     p_17 - 0/ </pre> </div></div> <p> this is correct output, llog has only one record #7707, but its bitmap has also bit #8245 set, that is why count is 2 and that is why retry doesn't help, it reads the same bitmap and llog data again and again. I will think how to inject such corruption for test purposes and will add modified llog_reader as well</p> <p>"Mike Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47934" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47934</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> llog: llog_reader to detect more corruptions<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e6f90aa1b8234121d0fc03ce12f98268ff3fcd29</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47698/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47698/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> lod: prevent endless retry in recovery thread<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 1a24dcdce121787428ea820561cfa16ae24bdf82</p> <p>"Mike Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48112" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48112</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> llog: more checks in llog_reader<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 4b619366a07813394cfb7abf1d79bb9512605401</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47934/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47934/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> llog: llog_reader to detect more corruptions<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d914a5b7a49ac6b61c0191a0966d1f684a6957b6</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48286" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48286</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> lod: prevent endless retry in recovery thread<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 7e2533728b5d574ec8638742cbfe574580c0a063</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48310" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48310</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> llog: llog_reader to detect more corruptions<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 02ea0e325eabc57d95051e79ffe1cc87c2243ced</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48341" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48341</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> llog: Fix chunk re-read case in llog_process_thread<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a7015dccd3e960516c95510663626f075191d4bd</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48112/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48112/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> llog: more checks in llog_reader<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 386ffcdbb4c9b89f798de4c83a51a3f020542c8b</p> <p>All patches seem to have merged for 2.16</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51217" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51217</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> lod: prevent endless retry in recovery thread<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: d66b517c9207dae3dd6da75266e78e50dfbc3f93</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51218" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51218</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> llog: llog_reader to detect more corruptions<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: e43bf0086e8f80f128cf868b5dca6079872f6a62</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51220" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51220</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15938" title="MDT recovery did not finish due to corrupt llog record" class="issue-link" data-issue-key="LU-15938"><del>LU-15938</del></a> llog: more checks in llog_reader<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: a0d25b76f6d41e164536a1c1cd46d503338643e7</p> Related LU-15937 LU-15761 LU-15645 LU-15934 LU-15139 LU-16203 LU-16052 Development Rank 1|i02ryf: Rank (Obsolete) 9223372036854775807 Severity