https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16159 Lustre <p>Once recovery is aborted, the existing update logs should be removed, because they are used for recovery only, and if it's corrupt, or inaccessible, if they are kept there after recovery abort, the next recovery will meet with the same issue again, besides, the log file size may become large, retrieving them in recovery may lead to recovery timeout.</p> <p>Besides this, in LOD device initialization, it should do some sanity check for all update logs on it, if the log file is not accessible (e.g. OI mapping mismatch, which may lead to recovery timeout too), delete the log file FID from update catalog (do not delete this file at the moment because it's inaccessible) so that the log file is not visible to others.</p> LU-16159

remove update logs after recovery abort

Bug Minor Reopened Unresolved Lai Siyao Lai Siyao Wed, 14 Sep 2022 14:26:49 +0000 Mon, 8 Jan 2024 03:32:52 +0000 Lustre 2.16.0 0 13 <p>I think that this is important and will resolve quite a number of problems that we've seen with DNE recovery. Currently, if there is <b>any</b> problem with the recovery llog, it will be kept and retried on every restart, and eventually grows to be very large and causes MDT recovery to be slow and/or fail repeatedly.</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/48584" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/48584</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> lod: remove update llogs upon abort_recovery<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 2<br/> Commit: 61697dd867a8a8933f3843fe5977de72ec3be91d</p> <p><del>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49052" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49052</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> test: debug patch</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: de62d8bb542bfb417422e42671967531c2370dc1</del></p> <p><del>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch:</del> <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49059" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49059</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> lod: debug patch 2</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 109c9b8487b800004a4e849aca1a9ad1a3eab6df</del></p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48584/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48584/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> lod: cancel update llogs upon recovery abort<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b054fcd7852f6a22f8ec469ce94ddf6f3331ab34</p> <p>Landed for 2.16</p> <p>MDSCOUNT=2 ONLY=100 MDSSIZE=400000 OSTSIZE=1000000 OSTCOUNT=2 REFORMAT=yes REFORMAT=yes bash replay-single.sh</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> == replay-single test complete, duration 135 sec ========= 13:47:49 (1671025669) rm: cannot remove <span class="code-quote">'/mnt/lustre/d100c.replay-single'</span>: Directory not empty replay-single test_135: @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = ./../tests/test-framework.sh:6526:error() = ./../tests/test-framework.sh:6010:check_and_cleanup_lustre() = replay-single.sh:5043:main() </pre> </div></div> <p>checked few commits around:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> COMMIT TESTED PASSED FAILED COMMIT DESCRIPTION 624e78ae80 1 0 1 BAD LU-930 docs: add lfs-rm_entry.8 man page d1dbf26afd 1 0 1 BAD LU-16291 build: make kobj_type constant 6f74bb60ff 1 0 1 BAD LU-16205 sec: reserve flag <span class="code-keyword">for</span> fid2path <span class="code-keyword">for</span> encrypted files b054fcd785 1 0 1 BAD LU-16159 lod: cancel update llogs upon recovery abort 1819f6006f 5 5 0 GOOD LU-15801 ldiskfs: Server support <span class="code-keyword">for</span> RHEL9 88bccc4fa4 5 5 0 GOOD LU-16114 build: Update security_dentry_init_security args </pre> </div></div> <p>In replay-single.sh formatall is called before check_and_cleanup_lustre:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 5040 (( $MDS1_VERSION &gt;= $(version_code 2.15.52.63) )) &amp;&amp; formatall 5041 5042 complete $SECONDS 5043 check_and_cleanup_lustre </pre> </div></div> <p>I pushed a patch under <a href="https://jira.whamcloud.com/browse/LU-16398" title="ost-pools: FAIL: remove sub-test dirs failed" class="issue-link" data-issue-key="LU-16398"><del>LU-16398</del></a> to disable this subtest until it is fixed. I think the right fix is to use rm_entry in a stack_trap at the end of test_100c, when rm_entry is fixed, and then more properly for LFSCK to actually make the directory "usable" again so that it can be removed with rmdir. </p> <p>Note that patch <a href="https://review.whamcloud.com/49335" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/49335</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16335" title="&quot;lfs rm_entry&quot; failed to remove broken directories" class="issue-link" data-issue-key="LU-16335"><del>LU-16335</del></a> test: add fail_abort_cleanup()</tt>" is already adding the stack_trap cleanup of the bad directory, so the only thing still needed after that is LFSCK to fix it properly (<a href="https://jira.whamcloud.com/browse/LU-16336" title="LFSCK should fix inconsistencies caused by recovery abort" class="issue-link" data-issue-key="LU-16336">LU-16336</a>).</p> <p>also, I noticed number of assertions I haven't seen few quite a while:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 6613.485294] Lustre: DEBUG MARKER: == replay-single test 119: timeout of normal replay does not cause DNE replay fails ========================================================== 12:56:09 (1671108969) ... [ 6692.155879] Lustre: 300726:0:(ldlm_lib.c:2305:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout [ 6692.156118] Lustre: 300726:0:(ldlm_lib.c:2315:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 6692.156264] Lustre: 300726:0:(ldlm_lib.c:2315:target_recovery_overseer()) Skipped 2 previous similar messages [ 6692.180534] Lustre: lustre-MDT0000-osd: cancel update llog [0x2000320e0:0x1:0x0] [ 6692.242056] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x24000c369:0x1:0x0] [ 6692.276833] Lustre: 300726:0:(ldlm_lib.c:2859:target_recovery_thread()) too <span class="code-object">long</span> recovery - read logs [ 6692.282517] LustreError: 7048:0:(dt_object.h:2296:dt_declare_record_write()) ASSERTION( dt-&gt;do_body_ops ) failed: [0x24001212a:0x2:0x0] doesn't exit [ 6692.282752] LustreError: 7048:0:(dt_object.h:2296:dt_declare_record_write()) LBUG [ 6692.282914] Pid: 7048, comm: ll_ost_out00_00 4.18.0 #2 SMP Sun Oct 23 17:58:04 UTC 2022 [ 6692.283111] Call Trace TBD: [ 6692.283190] [&lt;0&gt;] libcfs_call_trace+0x67/0x90 [libcfs] [ 6692.283312] [&lt;0&gt;] lbug_with_loc+0x3e/0x80 [libcfs] [ 6692.283488] [&lt;0&gt;] out_write_add_exec+0x175/0x1e0 [ptlrpc] [ 6692.283652] [&lt;0&gt;] out_write+0x161/0x380 [ptlrpc] [ 6692.283810] [&lt;0&gt;] out_handle+0x16c0/0x23b0 [ptlrpc] [ 6692.283970] [&lt;0&gt;] tgt_request_handle+0x977/0x1a40 [ptlrpc] [ 6692.284121] [&lt;0&gt;] ptlrpc_main+0x1724/0x32c0 [ptlrpc] [ 6692.284243] [&lt;0&gt;] kthread+0x129/0x140 </pre> </div></div> <p>It looks we shouldn't cancel update logs if some update logs have been replayed, because it may involve update log write, while update log cancel may destroy empty logs. I will push a patch to move update log cancel before request/update replay.</p> <p>Lai, it isn't clear if we can control when <tt>abort_recov_mdt</tt> is run (before or after recovery is done), because administrator may run this command tens of minutes after the MDS has restarted and recovery is hung, so it is likely that update log recovery is already finished (as much as possible) at this point.</p> <p>I looked through the Maloo results, and did see two crashes that look like <a href="https://jira.whamcloud.com/browse/LU-15139" title="sanity test_160h: dt_record_write() ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write ) failed" class="issue-link" data-issue-key="LU-15139"><del>LU-15139</del></a> "<tt>dt_record_write() ASSERTION( dt-&gt;do_body_ops-&gt;dbo_write )</tt>", per Alex's comment above. Also, patch <a href="https://review.whamcloud.com/49335" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/49335</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16335" title="&quot;lfs rm_entry&quot; failed to remove broken directories" class="issue-link" data-issue-key="LU-16335"><del>LU-16335</del></a> test: add fail_abort_cleanup()</tt>" v1 crashed 4 times with this LASSERT and I couldn't see any functional difference between the v1 of the patch and the current v3, nor in the parent patch <a href="https://review.whamcloud.com/49329" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/49329</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16335" title="&quot;lfs rm_entry&quot; failed to remove broken directories" class="issue-link" data-issue-key="LU-16335"><del>LU-16335</del></a> mdt: skip target check for rm_entry</tt>".</p> <p>Other replay-single crashes in the past few days can be attributed to the other patches themselves.</p> <p>Yes, the abort_recov_mdt can be run at any time, but update log cancel will only be done before request/update/ replay, which means, if request/replay has started, abort_recv_mdt won't cancel update logs.</p> <p>But won't that defeat the whole purpose of the fix? Could something else be done, like forcing rollover to new logs and then cancelling the old logs, so that in-use logs are not removed?</p> <p>"Lai Siyao &lt;lai.siyao@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49526" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49526</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> target: race in update log cancel<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 983a53137bfd26e6baaba4575a9dbb379e358b76</p> <p>Reopening this so that it is being tracked for 2.16 due to latest patch.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49633" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49633</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> tests: cleanup replay-single code style<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cb4e810ddfb11aab30ea6ad6b40ff371c04ddac1</p> <p>"Alex Zhuravlev &lt;bzzz@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49787" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49787</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> osp: destroy should not overtake writes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0e0993837d0e13bd31d53daea04e7259be6c1c4c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/49787/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/49787/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> osp: destroy should not overtake writes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5a5bd5b4dafaf252c641b8afd2cd809de7384f4f</p> <p>Is there anything left for this ticket, or can it be resolved?</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51223" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51223</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> lod: cancel update llogs upon recovery abort<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: a544f6c69fe4b2eb81f4b05581654325ecc96f93</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/51224" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/51224</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16159" title="remove update logs after recovery abort" class="issue-link" data-issue-key="LU-16159">LU-16159</a> osp: destroy should not overtake writes<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: acfee963c6999785afdd3b25c83015695099a3b9</p> Related LU-16336 LU-15934 LU-16335 LU-16398 LU-17365 Development Rank 1|i03053: Rank (Obsolete) 9223372036854775807 Severity