[LU-14338] prevent duplicate client mounts and server connections

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.12.5
• Type: Bug
• Priority: Minor
• Reporter: Nathan Dauchy (Inactive)
• Assignee: WC Triage
• Resolution: Unresolved

Environment

CentOS-7.7, lustre-2.12.5_ddn14

• Severity: 3

Description

The lustre client mount process should check whether another mount was already initiated, but not yet completed, for the specified filesystem name and mountpoint. If found, it should exit with a clear error message. It may help system administrators to debug mount problems if it also prints the duration of the prior attempt, in seconds.

The simple check of /etc/mtab leaves too large an opportunity for race condition, but the error message is helpful and can be used as an example:

# mount -t lustre 192.168.1.101@o2ib:192.168.1.102@o2ib:/lfstmp /mnt/lfstmp
mount.lustre: according to /etc/mtab 192.168.1.102@o2ib:192.168.1.101@o2ib:/lfstmp is already mounted on /mnt/lfstmp

So the new error message could look something like:

mount.lustre: $SERVER:/$FS is already mounting on $MNTPNT in progress for $SEC seconds

The problem can be "duplicated" fairly consistently on a client with:

# mount -t lustre 192.168.1.102@o2ib:/lfstmp /mnt/lfstmp & mount -t lustre 192.168.1.102@o2ib:/lfstmp /mnt/lfstmp
# mount -t lustre
192.168.1.102@o2ib:/lfstmp on /mnt/lfstmp type lustre (rw,flock,lazystatfs)
192.168.1.102@o2ib:/lfstmp on /mnt/lfstmp type lustre (rw,flock,lazystatfs)

I can think of at least 3 scenarios which would exacerbate the problem by extending the window where this can occur...

• Lustre modules are not yet loaded, so the first mount takes a second or two longer as it triggers the load and possibly even lnet initialization
• A network problem causes the first mount to hang, and an admin or self healing script tries again, then both mounts proceed when the network connectivity is restored. The "network problem" can be as simple as a client IB port not yet initialized on boot.
• In a failover configuration, the primary MGS NID is not available, so a first mount will hang until it proceeds to the secondary MGS NID after a timeout. This last case leads to a most reliable reproducer with a longer window for the race condition:

  # mount -t lustre 192.168.1.101@o2ib:192.168.1.102@o2ib:/lfstmp /mnt/lfstmp & sleep 20; mount -t lustre 192.168.1.102@o2ib:192.168.1.101@o2ib:/lfstmp /mnt/lfstmp; wait; mount -t lustre
  

In some (all?) cases, the duplicate mounts will be accompanied by multiple connections to each storage target, as shown by 'lctl dl'.

There are many possible reasons why mount would be run twice in the real world, and although a clean client configuration and good sysadmin practices may avoid them, Lustre itself should handle the problems gracefully with a simple error message, and prevent unnecessary connections to the servers.