[LU-5662] mkfs.lustre --replace can cause index numbers to be locked out

• Type: Bug
• Resolution: Low Priority
• Priority: Minor
• Affects Version/s: Lustre 2.5.3
• Environment: RHEL 6.5, Lustre 2.5.3
• Severity: 3
• Assignee: John Fuchs-Chesney (Inactive)
• Reporter: Alexander Mitin

Description

Running mkfs.lustre with the --replace flag causes a difficult-to-untangle lockout situation if the lustre MDS has never seen the index before.

I stumbled across this while scripting the reformatting operations of a test system. In this process, I ended up reformatting a completely new OST for the first time, using the --replace flag.

When I did this, the attempt to mount the OST on the OSS resulted in the following failure message:

mount.lustre: mount /dev/sdae at /lustre/ost6 failed: No such file or directory
Is the MGS specification correct?
Is the filesystem name correct?

Once this has happened, the MDS seems to be stuck in a halfway state, and there is no easy way recover use of the index number. If you reformat (again) with --replace specified, the error persists. If you reformat (again) with --replace absent, you get the following error on mount:

mount.lustre: mount /dev/sdac at /lustre/ost4 failed: Address already in use
The target service's index is already in use. (/dev/sdac)

Workaround is to dismount all Lustre disks, shut down lustre with lustre_rmmod, and then remount all disks.

This is similar to the -U versus -i flag in the rpm command. Technically, -U should not be used if the RPM has never been installed before, but the rpm command is intelligent enough to interpret -U as meaning -i if there is no prior installed version of the RPM. What rpm should certainly NOT do is to treat a technical misuse of -U on an uninstalled module as a persistent error that prevents subsequent use of either -U or -i on that module without a full reboot of Linux.

There are numerous ways to fix this. The simplest from a user perspective is this: if a formatted OST is flagged with --replace, then when mounting, the MDS should recognize it has never seen the index before, and simply ignore the --replace tag. This would be the analogue of rpm seeing the -U flag, recognizing that it has never seen the RPM before, and simply treating -U as -i.