[LU-16738] Improve mount.lustre with many MGS NIDs

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.16.0, Lustre 2.15.2
• Type: Improvement
• Priority: Minor
• Reporter: Andreas Dilger
• Assignee: James A Simmons
• Resolution: Unresolved

Sub-Tasks

Key	Summary	Type	Status	Assignee
LU-17379	try MGS NIDs more quickly at initial ...	Technical task	In Progress	Mikhail Pershin

• Key: Summary
• Summary: Type
• Type: Status
• Status: Assignee

Description

When the MGS has multipe failover hosts, and each host has multiple NIDs, the mount command line can become excessively long:

# mount -t lustre 172.16.0.26@o2ib,172.16.0.27@o2ib:172.16.0.30@o2ib,172.16.0.31@o2ib:172.16.0.28@o2ib,172.16.0.29@o2ib:172.16.0.24@o2ib,172.16.0.25@o2ib:/testfs /lustre/testfs/client

Not only is this very inconvenient to type and prone to error, but it is also unsightly in "df" and "mount" output because of the very long "device" name. This will become significantly worse once IPv6 NIDs are in use (each addresse will be about twice as long and hard to visually parse), and may even limit the number of NIDs that can be specified on the command-line due to the length.

It is already possible (though uncommonly used, for some reason) to use DNS hostname lookup for the MGS NID in the case of TCP hostnames. The mount.lustre command will do DNS hostname resolution and pass the numeric NIDs to the kernel:

# mount -t lustre mgs-fs1-1a@o2ib,mgs-fs1-1b@oi2b:mgs-fs1-2a@o2ib,mgs-fs1-2b@o2ib:mgs-fs1-3a@o2ib,mgs-fs1-3b@oi2b:mgs-fs1-4a@oi2b,mgs-fs1-4b@o2ib:/testfs /lustre/testfs/client

This is an improvement, but still requires that all of the MGS hostnames are supplied at mount, but at least avoids the need for manually specifying all of the IP addresses. If the network type is not specified, then it defaults to @tcp0, but it is possible to use @o2ib since this will also resolve to an IP address.

A few improvements could be made to shorten this command-line:

• allow host or NID ranges to be specified:

  # mount -t lustre 172.16.0.[24-30/2]@o2ib,172.16.0.[25-31/2]@o2ib:/testfs /lustre/testfs/client
  # mount -t lustre mgs-fs1-[1-4]a@o2ib,mgs-fs1-[1-4]b@o2ib:/testfs /lustre/testfs/client
  

  This will be significantly less to type than the full NID list, but is somewhat opaque and would still need to be expanded internally by mount.lustre to the full NID list for passing to the kernel. The full list would still be visible in the "mount" and "df" output.

• allow round-robin DNS for the MGS interfaces from a single hostname would avoid the need to specify multiple MGS hostnames on the command-line:

  # host mgs-fs1
  mgs-fs1 has address 141.193.213.10
  mgs-fs1 has address 141.193.213.11
  # mount -t mgs-fs1@o2ib:/testfs /lustre/testfs/client
  

  This should work for both IPv4 and IPv6 hostname resolution, but requires that mount.lustre and/or libcfs_str2nid() are able to resolve a single name to multiple NIDs.

All of the "mount.lustre" improvements only shorten the input name given on the command line. However, /proc/mounts would still show the full list of MGS NIDs for df and mount output.

One (possibly controversial) option would be to shorten the command-line displayed in /proc/mounts to only list the NID(s) of the currently-active MGS. That would keep the command-line down to ~2 NIDs, but would mean that copying the shortened displayed command-line to other nodes would potentially cause them problems if the MGS failed over, because they would only know one set of NIDs.

Another option would be (in the case of hostnames or round-robin DNS) would be to internally pass the original MGS name as an "MGS display name" as an option the kernel mount(2) command (e.g. "--mgs-display=mgs-cli1@o2ib"). This would be shown by /proc/mounts and should work equally well for other clients, since they should also be able to resolve the hostname similarly.

Issue Links

• is related to: LU-10391 LNET: Support IPv6
• is related to: LU-17379 try MGS NIDs more quickly at initial ...
• is related to: LU-16722 MGS config log restructuring and redu...
• is related to: LU-17186 Replace deprecated gethostby*() calls...

Comments

[Andreas Dilger]

In LU-17186 Jian wrote:

The getaddrinfo() function allocates and initializes a linked list of addrinfo structures, one for each network address that matches node and service, subject to any restrictions imposed by hints, and returns a pointer to the start of the list in res. The items in the linked list are linked by the ai_next field.

       int getaddrinfo(const char *node, const char *service,
                       const struct addrinfo *hints,
                       struct addrinfo **res);

       struct addrinfo {
           int              ai_flags;
           int              ai_family;
           int              ai_socktype;
           int              ai_protocol;
           socklen_t        ai_addrlen;
           struct sockaddr *ai_addr;
           char            *ai_canonname;
           struct addrinfo *ai_next;
       };

This paves the way to use this interface to fetch multiple addresses for a single MGS hostname on the mount.lustre command line.

I think the other thing that could be done is adding mount option with the "display name" for the MGS in the /proc/mounts output, which is the symbolic name to use for a client mount, even though it is not exactly the list of NIDs actually passed down to the kernel for mounting. We could have a separate llite.FSNAME-*.mount parameter that prints the actual mount command line for debugging if needed, but this wouldn't make "df" so ugly.