https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8361 Lustre <p>It appears that in lustre 2.8.0, the lctl lfsck_start --all command is broken:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@jet1:~]# lctl lfsck_start -A Must specify device to start LFSCK. [root@jet1:~]# lctl lfsck_start --all Must specify device to start LFSCK. </pre> </div></div> <p>But the man page says (in the lfsck section):</p> <blockquote> <p> -A, --all<br/> Start LFSCK on all available MDT devices.</p></blockquote> LU-8361

lctl lfsck_start --all does not start lfsck on all devices

Bug Major Resolved Fixed nasf Christopher Morrone llnl Fri, 1 Jul 2016 21:28:43 +0000 Tue, 28 Feb 2017 16:46:32 +0000 Thu, 8 Sep 2016 03:34:47 +0000 Lustre 2.8.0 Lustre 2.9.0 0 5 <p>Fan Yong</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Sorry for the confused man page. As you can see via "lctl lfsck_start --help"</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl lfsck_start --help start LFSCK usage: lfsck_start &lt;-M | --device {MDT,OST}_device&gt; [-A | --all] [-c | --create_ostobj [on | off]] [-C | --create_mdtobj [on | off]] [-e | --error {continue | abort}] [-h | --help] [-n | --dryrun [on | off]] [-o | --orphan] [-r | --reset] [-s | --speed ops_per_sec_limit] [-t | --type check_type[,check_type...]] [-w | --window_size size] options: ... </pre> </div></div> <p>The "&#45;M|&#45;&#45;device" option must be specified for any LFSCK command except "help" case. The "&#45;A|&#45;&#45;all" option is used for DNE case. If there are multiple MDTs in your system, then for the default mode, the LFSCK will be started on the specified MDT and related OSTs (for layout LFSCK case). If you want to start the LFSCK on the whole system, you need to specify both the "&#45;M|&#45;&#45;device" and the "&#45;A|&#45;&#45;all" options, then the specified MDT will dispatch the LFSCK command to other MDTs automatically.</p> <p>You may want to ask why must specify the "&#45;M|&#45;&#45;device" option, there are two main reasons:<br/> 1) It is not easy to decide the device name by the "lctl" command, Lustre supports multiple Lustre instances share the same physical node.<br/> 2) Historical compatibility consideration. As you know, we have several LFSCK releases since Lustre-2.3, but the full LFSCK support for DNE is released via Lustre-2.7. Before that all the LFSCK commands need the "&#45;M|&#45;&#45;device" option. Keeping such option will easy the existing LFSCK users/scripts.</p> <p>I don't buy the explanation that it is not easy to decide the device name from the lctl command. If there are multiple Lustre instances, you could simply start lfsck check on all of them. After all, it is the "all" command. If you want to make it easier for people to select one filesystem and not another, allow them to specify a filesystem name rather than a device name.</p> <p>The historical consideration doesn't really make too much sense to me either. You can support the older -M option without making it a required parameter in all instances.</p> <p>Yes, the man page definitely needs improvement as well.</p> <p>Here, the "Start LFSCK on all available MDT devices" wants to say that the user can start the LFSCK on all the available MDTs via single "lctl lfsck_start" command, instead of "lctl lfsck_start" on each MDT one by one. It is NOT exclusive with the "-M" option.</p> <p>As for the solution of specifying the filesystem name, on some degree, that is equal to split the "-M" option into several ones: "filesystem name" + "device role" + "device index", it is not impossible, but means more options. It may be convenient for some cases, but may be inconvenient for other cases. At the time of the first LFSCK released in Lustre-2.3, the "-M" option was enough, then the subsequent LFSCK releases inherited that.</p> <p>Anyway, I will update the man page to describe the things more clear to avoid misguiding.</p> <p>Here is the patch:<br/> <a href="http://review.whamcloud.com/#/c/19294/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/19294/</a></p> <blockquote><p>Here, the "Start LFSCK on all available MDT devices" wants to say that the user can start the LFSCK on all the available MDTs via single "lctl lfsck_start" command, instead of "lctl lfsck_start" on each MDT one by one. It is NOT exclusive with the "-M" option.</p></blockquote> <p>I understand that. I am saying that this design is not user friendly. I would love for it to work like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ lctl lfsck start </pre> </div></div> <p>Note that there is no underscore connecting "lfsck" and "start". And this would just start lfsck for all devices found for all filesystems found. Probably 99% of Lustre installations will not have multiple filesystems served from the same servers, so we optimize for the common case and make that clean and simple. Then we can add options to allow finer control for the remaining 1% of use cases.</p> <p>-M, which looks like and option but isn't optional, is not a clean user interface design. And I don't agree that it is functionally equivalent to a filesystem name. A sysadmin is more likely to know the filesystem's name off the top of their head. An individual MDT name that is valid on the current node is probably something that they will always need to look up. And what is the command to look that up? "lctl dl"? That command is terrible.</p> <p>Come to think of it, rather than making the filesystem name an option, and because single filesystems are the most common, you could just make "lctl lfsck start" return an error if there is more than one filesystem. Something like this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ lctl lfsck start Error: more than one filesystem, please specify which one: lquake ltest $ lctl lfsck start lquake Starting lfsck on lquake </pre> </div></div> <p>But on a system with only one filesystem it would just do:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ lctl lfsck start Starting lfsck on bigfs </pre> </div></div> <p>Chris,<br/> please note that <tt>-M &lt;fsname&gt;</tt> does not need the full device UUID, just the target name like <tt>&lt;fsname&gt;-MDT0000</tt> so it does not need to be looked up by <tt>lctl dl</tt> each time. That said, I agree there could be improvements in this area.</p> <p>Fan Yong,<br/> I don't think it would be a problem to allow <tt>lctl lfsck_start -A</tt> to run on all filesystems (MDTs and OSTs) that are connected to the current node. Since all of the current users would be using <tt>-M &lt;mdt_name&gt;</tt>, and just not requiring that option together with <tt>-A</tt> would keep compatibility and also address Chris' concerns. It should also be relatively straight forward for <tt>-A -M &lt;fsname&gt;</tt> to allow just specifying the fsname, which is no less specific than using <tt>-A -M &lt;fsname&gt;-MDT0000</tt> since it is already checking all the devices in that filesystem. The device name(s) list can be found with something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> glob_t param; rc = llapi_get_param_path(NULL, <span class="code-quote">"mdd.*-MDT0000"</span>, FILTER_BY_EXACT, NULL, &amp;param); </pre> </div></div> <blockquote><p>please note that -M &lt;fsname&gt; does not need the full device UUID, just the target name like &lt;fsname&gt;-MDT0000 so it does not need to be looked up by lctl dl each time</p></blockquote> <p>I know. But there are now more targets than just 0000. It requires a lookup or some other external knowledge to know the mapping of MDTs to MDS nodes. It is an added difficulty that is easily avoided with good command design.</p> <blockquote><p>rc = llapi_get_param_path(NULL, "mdd.*-MDT0000", FILTER_BY_EXACT, NULL, &amp;param);</p></blockquote> <p>It can't be a literal "0000". Not all MDS nodes will be serving that particular index.</p> <p>I was thinking that "lctl lfsck_start -A" would be run on the primary MDS and then it doesn't matter where the rest of the MDTs are located since "-A" will start lfsck on all of the other MDS and OSS nodes. </p> <p>I would prefer not to make that restriction if it does not already exist. From a usability standpoint, it would be nice if admins could start lfsck from the MGS too, and maybe even from clients in the future.</p> <p>If any restrictions about where the command can be run, it needs to be clearly stated.</p> <p>And if the command can only be run on the node with the zero index MDT, then the error message needs to be very clear when the command is run on a node without that MDT.</p> <p>I'm fine with allowing lfsck to be started on any MDT if that isn't already a restriction (I've always started it on MDT0000 by virtue of only having one MDS on my home system). I'm against allowing administrator tasks to be run from clients, since this adds an extra level of security issues, and IMHO doesn't provide any significant benefit. </p> <p>The current implementation supports to start the LFSCK on any MDT as long as you specify the "-M" option properly, for example "-M ${fsname}-MDT0001" also works. But it does NOT allow to be started from client, nor from a separated MGS node.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/21596" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21596</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8361" title="lctl lfsck_start --all does not start lfsck on all devices" class="issue-link" data-issue-key="LU-8361"><del>LU-8361</del></a> lfsck: detect Lustre device automatically<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5e2f4b5358e926ae20658f6e5ba313e253db32c0</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/21596/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/21596/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8361" title="lctl lfsck_start --all does not start lfsck on all devices" class="issue-link" data-issue-key="LU-8361"><del>LU-8361</del></a> lfsck: detect Lustre device automatically<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a0f7174c4106104f45977eeec7338e8f7fd1dafa</p> <p>The patch has been landed to master.</p> Related Development Rank 1|hzyggn: Rank (Obsolete) 9223372036854775807 Severity