[LU-16232] emergency llog cleanup server scripts

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.16.0
• Type: Improvement
• Priority: Minor
• Reporter: Mikhail Pershin
• Assignee: Mikhail Pershin
• Resolution: Fixed

Description

There can be situations when update llog or changelog files are corrupted and we are just removing/truncating them. Scripts are needed to remove corrupted llogs properly with all plain llogs, so no orphaned data remains on server

Comments

[Mikhail Pershin]

Copied from EX-4969:

For both scripts the steps to cleanup problematic llogs are:

• mount MDT filesystem locally on server as ldiskfs mount
• run script first in dry-run mode to make sure it parses llogs as needed and that all needed tools are in place:

 # bash remove_(changelog|updatelog) -n <ldiskfs_mount>

• run it to save all llogs for analysis

# bash remove_(changelog|updatelog) -n -z /tmp/llogs_saved <ldiskfs_mount>

• to be sure check that /tmp/llogs_saved.tar.gz (or whatever name prefix you've used) exists and has all llogs inside:

# ls -ali /tmp/llogs_saved.tar.gz
# tar -tf /tmp/llog_saved.tar.gz

• finally run script to delete all llogs:

 # bash remove_(changelog|updatelog) <ldiskfs_mount>

Note: for better llogs compression xz can be used as well, pass it to the script via GZIP env variable:

# GZIP=xz bash remove_(changelog|updatelog) -n -z /tmp/llogs_saved <ldiskfs_mount>

Archive name will ends with .xz in that case instead of .gz

[Gerrit Updater]

"Mikhail Pershin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/48838
Subject: LU-16232 scripts: changelog/updatelog emergency cleanup
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 9129cea32def94999b67e942c69d0a5517613fd3

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/48838/
Subject: LU-16232 scripts: changelog/updatelog emergency cleanup
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b533700add91fe4220f50d057a470e0b6f4893c9

[Peter Jones]

Landed for 2.16

[Andreas Dilger]

In further discussion, it would be useful to allow a "-m <mdt_index>" argument to the script to have it selectively delete only the update logs for the specified MDT index. Otherwise, if the script is run on a remote MDT, then it will delete the update logs for all other MDTs, and then those MDTs will then have issues during operation and/or later recovery.

[Chris Hunter (Inactive)]

I receive an error message when using the script:

./remove_updatelog.lu16232.sh: line 100: syntax error near unexpected token `<'
./remove_updatelog.lu16232.sh: line 100: `        read -r -d '' -a OPT_MDTS < <(hexdump -v -e '2/8 " %16x" 2/8 "\n"' $catlist |'

appears to be an issue with shell pipe
eg)

read -r -d '' -a OPT_MDTS < <(hexdump -v -e '2/8 " %16x" 2/8 "\n"' $catlist | awk '{print "[0x"$2":0x"$1":0x0]"}')

instead of

read -r -d '' -a OPT_MDTS <<< $(hexdump -v -e '2/8 " %16x" 2/8 "\n"' $catlist | awk '{print "[0x"$2":0x"$1":0x0]"}')

[Mikhail Pershin]

Chris, try it with bash : # bash ./remove_updatelog

The syntax of '< <()' correct in bash (that is process substitution, creates temp file and redirects from it) but may be not in other shells. Your proposal is possible replacement so it is worth to update script in that manner but for now just call bash explicitly

[Gerrit Updater]

"Mikhail Pershin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50558
Subject: LU-16232 scripts: clean specific MDTs update llogs
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a399335cdcc6561077a1400572128b523c461a16

[Mikhail Pershin]

I've just added to the ticket another script which allows to collect update logs on live server node by using debugfs. It helps to get llogs for further analysis and replace multiple commands need to be executed for that

[Gerrit Updater]

"Yang Sheng <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50876
Subject: LU-16232 script: fix the argument parse
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: aa0ded5d3504187b1f3fe852bb5e464569f5eb7c

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50558/
Subject: LU-16232 scripts: clean specific MDTs update llogs
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8546628c23b1ba1785e42ec9b1bc4b77acced5c6

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50876/
Subject: LU-16232 script: fix the argument parse
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 99144a595b767ef79acec058c838759bea73c579