Details
-
Technical task
-
Resolution: Fixed
-
Critical
-
Lustre 2.6.0
-
None
-
12614
Description
LFSCK needs to log with D_LFSCK all fixes that it makes and any inconsistencies that it finds that it does not repair (e.g. unknown LOV magic layouts). Otherwise it will be making secret changes to the filesystem and when there are problems they will be impossible to debug.
Also, if LFSCK is fining inconsistencies in the filesystem it is important for administrators and developers to know this in case they are being caused by bugs in the Lustre code or due to hardware or other problems.
I think logging inconsistencies (whether repaired or not) via D_LFSCK is the best way to do this. It would be useful to have a common format for such messages, so they could be parsed later to generate pathnames from the FIDs in the log messages. We should avoid pruning D_LFSCK messages that are not related to problems in the filesystem, except for limited start and stop messages. It might be necessary to quiet messages about adding individual FIDs to the OI file if the file was removed, since that will be too many messages on a large filesystem.
The D_LFSCK flag should be added to the default flags that are enabled all the time, since this should not normally generate any log messages on a correct filesystem (except limited start/stop messages).
Attachments
Issue Links
- is related to
-
LU-5202 LFSCK 5: LFSCK needs to log all changes and errors found
-
- In Progress
-
Activity
Patch to turn on LFSCK debug messages in our test infrastructure is at: http://review.whamcloud.com/#/c/10814
I have enhanced the patch http://review.whamcloud.com/#/c/10579/ to add more D_LFSCK message for LFSCK check/repair, and enable D_LFSCK by default for tests.
I think it also makes sense to enable D_LFSCK logging by default, once noisy messages are quieted (in patch 10579).
Since then, I prefer to split the work into two parts:
1) The first part will add/remove/change the D_LFSCK debug log in master, and the users can dump the LFSCK log via "lctl debug_kernel". Such patch should be landed to Lustre-2.6
2) The second part will implement some mechanism to start/stop/filter/collect the LFSCK logs automatically. Such patch will be landed to Lustre-2.7 or later.
I think at least some basic mechanism to log the repairs from LFSCK should be available in 2.6. Otherwise, there is no easy way for an administrator to find out what is wrong with the filesystem, even if --dry-run is used and they look in the logs.
On my home 2.4 filesystem (which was upgraded incrementally from 1.4), I've run "lctl lfsck_start -t namespace -M myth-MDT0000" repeatedly, and it reports in mdd.myth-MDT0000.lfsck_namespace:
checked_phase1: 3688766 checked_phase2: 222 updated_phase1: 1762863 updated_phase2: 147
even though I have enabled dirdata on the filesystem.
Looking into the b2_4 mdd_lfsck_namespace_exec_dir() it modifies the filesystem and increments the ln_linkea_repaired or ln_items_repaired counter in a few places, but doesn't print anything to D_LFSCK. This is still true in the master lfsck_namespace_exec_dir() code.
A similar situation exists in lfsck_namespace_double_scan_one() where "update = true" can be set (e.g. after lfsck_linkea_entry_unpack() removes a linkEA entry, but then "continue" is hit in the loop and nothing is printed. It looks like the LPF_DRYRUN check should be moved after the D_LFSCK message at the shrink: label (but before "update = true" is set) and maybe make it "continue" instead of "RETURN(1)" so that it will check all of the entries. It also seems in that a D_LFSCK message should be printed in lfsck_linkea_entry_unpack() if it drops an entry, and it should add a check for LFP_DRY_RUN.
I was testing with debug_daemon, but it seems this is logging all of the messages to the debug file, and then the lctl filter|show is only being used by lctl debug_file to filter the binary logs into ASCII at the end, so I don't know if this will be enough to be useful. I also tried lctl set_param printk=+lfsck to log all the D_LFSCK messages to the console, but this was much too verbose due to osd_otable_it_preload() printing a message for every inode being loaded.
I pushed http://review.whamcloud.com/10579 which just fixes the debug logging for osd_otable_it_preload(), but this is only the very start of what is needed for D_LFSCK logging to be useful.
Hi Jodi,
LFSCK logging mechanism is one of the supplementary functionalities to help the Lustre users to understand what the LFSCK repaired. It will cover all the LFSCK phases repairing, not only for MDT-OST consistency. Other LFSCK basic functionalities does not depends on the logging. So I do NOT think it is mandatory for Lustre-2.6, instead, We can do that in Lustre-2.7 or in LFSCK phase IV.
Andreas, what's your suggestion?
Fan Yong,
Could you please have a look at this one as it is being tracked for 2.6?
Thank you!
James, any progress on this issue? Have you been able to get debug_daemon started manually during LFSCK testing? I think it would be straight forward to add filtering capability to debug_daemon so it only dumps D_LFSCK messages to the log file.
Still one patch to land to turn the LFSCK debugging on by default.