https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16232 Lustre <p>There can be situations when update llog or changelog files are corrupted and we are just removing/truncating them. Scripts are needed to remove corrupted llogs properly with all plain llogs, so no orphaned data remains on server</p> LU-16232

emergency llog cleanup server scripts

Improvement Minor Resolved Fixed Mikhail Pershin Mikhail Pershin Tue, 11 Oct 2022 08:51:17 +0000 Thu, 31 Aug 2023 06:29:57 +0000 Tue, 30 May 2023 16:44:04 +0000 Lustre 2.16.0 0 10 <p>Copied from EX-4969:</p> <p>For both scripts the steps to cleanup problematic llogs are:</p> <ul> <li>mount MDT filesystem locally on server as ldiskfs mount</li> <li>run script first in dry-run mode to make sure it parses llogs as needed and that all needed tools are in place:</li> </ul> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> # bash remove_(changelog|updatelog) -n &lt;ldiskfs_mount&gt;</pre> </div></div> <ul> <li>run it to save all llogs for analysis</li> </ul> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># bash remove_(changelog|updatelog) -n -z /tmp/llogs_saved &lt;ldiskfs_mount&gt;</pre> </div></div> <ul> <li>to be sure check that <tt>/tmp/llogs_saved.tar.gz</tt> (or whatever name prefix you've used) exists and has all llogs inside:</li> </ul> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls -ali /tmp/llogs_saved.tar.gz # tar -tf /tmp/llog_saved.tar.gz</pre> </div></div> <ul> <li>finally run script to delete all llogs:</li> </ul> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> # bash remove_(changelog|updatelog) &lt;ldiskfs_mount&gt;</pre> </div></div> <p>Note: for better llogs compression <tt>xz</tt> can be used as well, pass it to the script via GZIP env variable:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># GZIP=xz bash remove_(changelog|updatelog) -n -z /tmp/llogs_saved &lt;ldiskfs_mount&gt;</pre> </div></div> <p>Archive name will ends with <tt>.xz</tt> in that case instead of <tt>.gz</tt></p> <p>"Mikhail Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48838" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48838</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16232" title="emergency llog cleanup server scripts" class="issue-link" data-issue-key="LU-16232"><del>LU-16232</del></a> scripts: changelog/updatelog emergency cleanup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 9129cea32def94999b67e942c69d0a5517613fd3</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/48838/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/48838/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16232" title="emergency llog cleanup server scripts" class="issue-link" data-issue-key="LU-16232"><del>LU-16232</del></a> scripts: changelog/updatelog emergency cleanup<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: b533700add91fe4220f50d057a470e0b6f4893c9</p> <p>Landed for 2.16</p> <p>In further discussion, it would be useful to allow a "<tt>-m &lt;mdt_index&gt;</tt>" argument to the script to have it selectively delete only the update logs for the specified MDT index. Otherwise, if the script is run on a remote MDT, then it will delete the update logs for <b>all other</b> MDTs, and then those MDTs will then have issues during operation and/or later recovery.</p> <p>I receive an error message when using the script:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>./remove_updatelog.lu16232.sh: line 100: syntax error near unexpected token `&lt;' ./remove_updatelog.lu16232.sh: line 100: `        read -r -d '' -a OPT_MDTS &lt; &lt;(hexdump -v -e '2/8 " %16x" 2/8 "\n"' $catlist |' </pre> </div></div> <p>appears to be an issue with shell pipe<br/> eg)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>read -r -d '' -a OPT_MDTS &lt; &lt;(hexdump -v -e '2/8 " %16x" 2/8 "\n"' $catlist | awk '{print "[0x"$2":0x"$1":0x0]"}') </pre> </div></div> <p>instead of</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>read -r -d '' -a OPT_MDTS &lt;&lt;&lt; $(hexdump -v -e '2/8 " %16x" 2/8 "\n"' $catlist | awk '{print "[0x"$2":0x"$1":0x0]"}') </pre> </div></div> <p>Chris, try it with bash : <tt># bash ./remove_updatelog</tt></p> <p>The syntax of '<tt>&lt; &lt;()</tt>' correct in bash (that is process substitution, creates temp file and redirects from it) but may be not in other shells. Your proposal is possible replacement so it is worth to update script in that manner but for now just call bash explicitly</p> <p>"Mikhail Pershin &lt;mpershin@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50558" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50558</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16232" title="emergency llog cleanup server scripts" class="issue-link" data-issue-key="LU-16232"><del>LU-16232</del></a> scripts: clean specific MDTs update llogs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a399335cdcc6561077a1400572128b523c461a16</p> <p>I've just added to the ticket another script which allows to collect update logs on live server node by using debugfs. It helps to get llogs for further analysis and replace multiple commands need to be executed for that</p> <p>"Yang Sheng &lt;ys@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50876" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50876</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16232" title="emergency llog cleanup server scripts" class="issue-link" data-issue-key="LU-16232"><del>LU-16232</del></a> script: fix the argument parse<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: aa0ded5d3504187b1f3fe852bb5e464569f5eb7c</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50558/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50558/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16232" title="emergency llog cleanup server scripts" class="issue-link" data-issue-key="LU-16232"><del>LU-16232</del></a> scripts: clean specific MDTs update llogs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8546628c23b1ba1785e42ec9b1bc4b77acced5c6</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50876/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50876/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16232" title="emergency llog cleanup server scripts" class="issue-link" data-issue-key="LU-16232"><del>LU-16232</del></a> script: fix the argument parse<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 99144a595b767ef79acec058c838759bea73c579</p> Related Development Rank 1|i032jj: Rank (Obsolete) 9223372036854775807