[LU-8361] lctl lfsck_start --all does not start lfsck on all devices

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.8.0
• Fix Version/s: Lustre 2.9.0
• Type: Bug
• Priority: Major
• Reporter: Christopher Morrone
• Assignee: nasf (Inactive)
• Resolution: Fixed
• Labels: llnl
• Severity: 3

Description

It appears that in lustre 2.8.0, the lctl lfsck_start --all command is broken:

[root@jet1:~]# lctl lfsck_start -A
Must specify device to start LFSCK.
[root@jet1:~]# lctl lfsck_start --all 
Must specify device to start LFSCK.

But the man page says (in the lfsck section):

-A, --all
Start LFSCK on all available MDT devices.

Comments

[Peter Jones]

Fan Yong

Could you please advise on this one?

Thanks

Peter

[nasf (Inactive)]

Sorry for the confused man page. As you can see via "lctl lfsck_start --help"

# lctl lfsck_start --help
start LFSCK
usage:
lfsck_start <-M | --device {MDT,OST}_device>
	     [-A | --all] [-c | --create_ostobj [on | off]]
	     [-C | --create_mdtobj [on | off]]
	     [-e | --error {continue | abort}] [-h | --help]
	     [-n | --dryrun [on | off]] [-o | --orphan]
             [-r | --reset] [-s | --speed ops_per_sec_limit]
             [-t | --type check_type[,check_type...]]
	     [-w | --window_size size]
options:
...

The "-M|--device" option must be specified for any LFSCK command except "help" case. The "-A|--all" option is used for DNE case. If there are multiple MDTs in your system, then for the default mode, the LFSCK will be started on the specified MDT and related OSTs (for layout LFSCK case). If you want to start the LFSCK on the whole system, you need to specify both the "-M|--device" and the "-A|--all" options, then the specified MDT will dispatch the LFSCK command to other MDTs automatically.

You may want to ask why must specify the "-M|--device" option, there are two main reasons:
1) It is not easy to decide the device name by the "lctl" command, Lustre supports multiple Lustre instances share the same physical node.
2) Historical compatibility consideration. As you know, we have several LFSCK releases since Lustre-2.3, but the full LFSCK support for DNE is released via Lustre-2.7. Before that all the LFSCK commands need the "-M|--device" option. Keeping such option will easy the existing LFSCK users/scripts.

[Christopher Morrone]

I don't buy the explanation that it is not easy to decide the device name from the lctl command. If there are multiple Lustre instances, you could simply start lfsck check on all of them. After all, it is the "all" command. If you want to make it easier for people to select one filesystem and not another, allow them to specify a filesystem name rather than a device name.

The historical consideration doesn't really make too much sense to me either. You can support the older -M option without making it a required parameter in all instances.

Yes, the man page definitely needs improvement as well.

[nasf (Inactive)]

Here, the "Start LFSCK on all available MDT devices" wants to say that the user can start the LFSCK on all the available MDTs via single "lctl lfsck_start" command, instead of "lctl lfsck_start" on each MDT one by one. It is NOT exclusive with the "-M" option.

As for the solution of specifying the filesystem name, on some degree, that is equal to split the "-M" option into several ones: "filesystem name" + "device role" + "device index", it is not impossible, but means more options. It may be convenient for some cases, but may be inconvenient for other cases. At the time of the first LFSCK released in Lustre-2.3, the "-M" option was enough, then the subsequent LFSCK releases inherited that.

Anyway, I will update the man page to describe the things more clear to avoid misguiding.

[nasf (Inactive)]

Here is the patch:
http://review.whamcloud.com/#/c/19294/

[Christopher Morrone]

Here, the "Start LFSCK on all available MDT devices" wants to say that the user can start the LFSCK on all the available MDTs via single "lctl lfsck_start" command, instead of "lctl lfsck_start" on each MDT one by one. It is NOT exclusive with the "-M" option.

I understand that. I am saying that this design is not user friendly. I would love for it to work like this:

$ lctl lfsck start

Note that there is no underscore connecting "lfsck" and "start". And this would just start lfsck for all devices found for all filesystems found. Probably 99% of Lustre installations will not have multiple filesystems served from the same servers, so we optimize for the common case and make that clean and simple. Then we can add options to allow finer control for the remaining 1% of use cases.

-M, which looks like and option but isn't optional, is not a clean user interface design. And I don't agree that it is functionally equivalent to a filesystem name. A sysadmin is more likely to know the filesystem's name off the top of their head. An individual MDT name that is valid on the current node is probably something that they will always need to look up. And what is the command to look that up? "lctl dl"? That command is terrible.

[Christopher Morrone]

Come to think of it, rather than making the filesystem name an option, and because single filesystems are the most common, you could just make "lctl lfsck start" return an error if there is more than one filesystem. Something like this:

$ lctl lfsck start
Error: more than one filesystem, please specify which one:
  lquake
  ltest
$ lctl lfsck start lquake
Starting lfsck on lquake

But on a system with only one filesystem it would just do:

$ lctl lfsck start
Starting lfsck on bigfs

[Andreas Dilger]

Chris,
please note that -M <fsname> does not need the full device UUID, just the target name like <fsname>-MDT0000 so it does not need to be looked up by lctl dl each time. That said, I agree there could be improvements in this area.

Fan Yong,
I don't think it would be a problem to allow lctl lfsck_start -A to run on all filesystems (MDTs and OSTs) that are connected to the current node. Since all of the current users would be using -M <mdt_name>, and just not requiring that option together with -A would keep compatibility and also address Chris' concerns. It should also be relatively straight forward for -A -M <fsname> to allow just specifying the fsname, which is no less specific than using -A -M <fsname>-MDT0000 since it is already checking all the devices in that filesystem. The device name(s) list can be found with something like:

        glob_t param;

        rc = llapi_get_param_path(NULL, "mdd.*-MDT0000", FILTER_BY_EXACT, NULL, &param);

[Christopher Morrone]

please note that -M <fsname> does not need the full device UUID, just the target name like <fsname>-MDT0000 so it does not need to be looked up by lctl dl each time

I know. But there are now more targets than just 0000. It requires a lookup or some other external knowledge to know the mapping of MDTs to MDS nodes. It is an added difficulty that is easily avoided with good command design.

rc = llapi_get_param_path(NULL, "mdd.*-MDT0000", FILTER_BY_EXACT, NULL, &param);

It can't be a literal "0000". Not all MDS nodes will be serving that particular index.

[Andreas Dilger]

I was thinking that "lctl lfsck_start -A" would be run on the primary MDS and then it doesn't matter where the rest of the MDTs are located since "-A" will start lfsck on all of the other MDS and OSS nodes.

[Christopher Morrone]

I would prefer not to make that restriction if it does not already exist. From a usability standpoint, it would be nice if admins could start lfsck from the MGS too, and maybe even from clients in the future.

If any restrictions about where the command can be run, it needs to be clearly stated.

And if the command can only be run on the node with the zero index MDT, then the error message needs to be very clear when the command is run on a node without that MDT.

[Andreas Dilger]

I'm fine with allowing lfsck to be started on any MDT if that isn't already a restriction (I've always started it on MDT0000 by virtue of only having one MDS on my home system). I'm against allowing administrator tasks to be run from clients, since this adds an extra level of security issues, and IMHO doesn't provide any significant benefit.

[nasf (Inactive)]

The current implementation supports to start the LFSCK on any MDT as long as you specify the "-M" option properly, for example "-M ${fsname}-MDT0001" also works. But it does NOT allow to be started from client, nor from a separated MGS node.

[Gerrit Updater]

Fan Yong ([e-mail]) uploaded a new patch: http://review.whamcloud.com/21596
Subject: LU-8361 lfsck: detect Lustre device automatically
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5e2f4b5358e926ae20658f6e5ba313e253db32c0

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/21596/
Subject: LU-8361 lfsck: detect Lustre device automatically
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a0f7174c4106104f45977eeec7338e8f7fd1dafa

[nasf (Inactive)]

The patch has been landed to master.