https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5202 Lustre <p>LFSCK needs to log with D_LFSCK all fixes that it makes and any inconsistencies that it finds that it does not repair (e.g. unknown LOV magic layouts). Otherwise it will be making secret changes to the filesystem and when there are problems they will be impossible to debug.</p> <p>There should be a mechanism for logging D_LFSCK messages to a separate log file for administrators to review, so that the kernel debug messages are not lost. A simple mechanism would be debug_daemon to log all messages to a file, then "lctl filter all; lctl show lfsck" (or similar) to filter all except lfsck messages into a text log file.</p> <p>The problem with this approach is that debug_daemon will consume all debug messages while it is running, and it will log a lot more than just D_LFSCK messages to disk. We may want to consider some other logging mechanism to capture just the D_LFSCK messages.</p> LU-5202

LFSCK 5: LFSCK needs to log all changes and errors found

Improvement Major In Progress Unresolved WC Triage Andreas Dilger mq115 Mon, 16 Jun 2014 18:06:20 +0000 Thu, 12 Aug 2021 21:40:50 +0000 Lustre 2.6.0 Lustre 2.5.5 0 12 <p>Why not use <a href="https://jira.whamcloud.com/browse/LU-4610" title="LFSCK needs to log all changes and errors found" class="issue-link" data-issue-key="LU-4610"><del>LU-4610</del></a>?</p> <p>Because <a href="https://jira.whamcloud.com/browse/LU-4610" title="LFSCK needs to log all changes and errors found" class="issue-link" data-issue-key="LU-4610"><del>LU-4610</del></a> needs to be closed for 2.6.0 to be released, and it isn't likely that this code will be landed for 2.6.0. </p> <p>On 2.5.3-ge835226 I ran <tt>lctl lfsck_start -M myth-MDT0000 -t namespace</tt> and enabled D_LFSCK printing to the console via <tt>lctl set_param printk=lfsck</tt>. This generated a lot of console messages that weren't very useful:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jan 4 21:36:33 mookie kernel: Lustre: 3402:0:(osd_scrub.c:1240:osd_otable_it_preload()) OSD pre-loaded: max = 2621440, preload = 2621438, rc = 0 Jan 4 21:36:33 mookie kernel: Lustre: 3402:0:(osd_scrub.c:1240:osd_otable_it_preload()) OSD pre-loaded: max = 2621440, preload = 2621439, rc = 0 Jan 4 21:36:33 mookie kernel: Lustre: 3402:0:(osd_scrub.c:1240:osd_otable_it_preload()) OSD pre-loaded: max = 2621440, preload = 2621440, rc = 0 Jan 4 21:36:33 mookie kernel: Lustre: 3402:0:(osd_scrub.c:1240:osd_otable_it_preload()) OSD pre-loaded: max = 2621440, preload = 2621441, rc = 1 Jan 4 21:36:33 mookie kernel: Lustre: 3403:0:(osd_scrub.c:1301:osd_scrub_main()) OI scrub: stop, rc = 1, pos = 2621441 Jan 4 21:36:33 mookie kernel: Lustre: 3402:0:(lfsck_engine.c:340:lfsck_master_engine()) LFSCK exit: oit_flags = 0xc0002, dir_flags = 0xc004, oit_cookie = 2130251, dir_cookie = 3397299287892376453, parent = [0x20814b:0x17d4596b:0x0], pid = 3402, rc = 1 </pre> </div></div> <p>This should be fixed for b2_5.</p> <p>For 2.7, we discussed adding <tt>lctl lfsck_start -v</tt> and <tt>-q</tt> options to enable the D_LFSCK messages to print to the console or not. Also, major messages like start/stop, urgent scrub, and first message of non-upgrade error should always be printed to the console via D_CONSOLE.</p> <p>In the meantime, it would be pretty useful to have a procfs tunable to dump lustre debug logs when an inconsistency is found by lfsck (similar to dump_on_eviction). It would help us debugging on lola.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13864" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13864</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5202" title="LFSCK 5: LFSCK needs to log all changes and errors found" class="issue-link" data-issue-key="LU-5202">LU-5202</a> lfsck: dump LFSCK debug log automatically<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 06777610a8e5b16cfb66cd0a54bc616e965c28d2</p> <p>I've just been using <tt>lctl set_param printk=+lfsck</tt> to have lfsck messages go to the console. That is ok with master, but b2_5 needs to be fixed so all the OI scrub status messages are not using D_LFSCK. </p> <p>The patch 13864 is used as temporary solution for lola test.</p> <p>The patch current has two negative reviews and has failed Maloo - we cannot use this on soak test in this state.<br/> Is there a proper fix that is going to land somewhere?</p> <blockquote> <p>The patch current has two negative reviews and has failed Maloo - we cannot use this on soak test in this state.<br/> Is there a proper fix that is going to land somewhere?</p></blockquote> <p>Sorry, there is not suitable patch yet, we need more consideration for that.</p> <p>While we are waiting for a revised patch and new feature... can you please clarify the procedure for gathering the logs and perhaps deciphering them?</p> <p>What I have been able to come up with so far is...</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lctl set_param printk=+lfsck lctl filter all_subs lctl show lfsck lctl clear lctl debug_daemon start /var/log/lfsck.debug lctl lfsck_start -A -M nbptest-MDT0000 -c on -C on -o lctl get_param -n osd-ldiskfs.*.oi_scrub | grep status lctl debug_daemon stop lctl set_param printk=-lfsck lctl debug_file /var/log/lfsck.debug | grep lfsck | egrep -v "(namespace LFSCK add flag| RPC pname)" &gt; /var/log/lfsck.log lctl show all_subs </pre> </div></div> <p>I'm not sure that the "filter" commands are actually doing anything useful though. And I would like to know if that method leaves out anything useful from the lfsck.log file.</p> <p>What you've got there is belt and suspenders. The printk setting will itself dump all lfsck messages to the console (dmesg &amp; /var/log/messages). If you don't want that, then debug_daemon is the alternative. The filter setting is indeed irrelevant since all the subsystems are already enabled by default. </p> Duplicate LU-8169 Related LU-4610 LU-6102 Development Rank 1|hzwp4n: Rank (Obsolete) 14526