-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
None
-
None
-
3
-
9223372036854775807
Problem
A target cannot use the network mount option with LNet dynamic peer discovery on. server_get_mti() and lmd_parse() reject the option with EINVAL when discovery is on. A target uses this option to keep traffic between targets on a dedicated network, away from tenant networks. A site that needs the option must turn discovery off on the servers. Discovery off does not scale on a production cluster with many peers.
The scope of this ticket is the target mount option. The client mount option is out of scope. A client mount with network and discovery on stays rejected.
Related failure
A client with discovery on can also fail to mount from servers with discovery off, when the client has two or more networks. The mount fails with -5. LNetAddPeer() adds the server NIDs as Multi-Rail peers. A ping reply shows that the server has discovery off, and discovery then deletes each peer. The next configuration record adds the peer again. Each delete can move the primary NID of the server, so the client loses replies.
Environment
- master, tag 2.17.61.
- Four rocky9 VMs: MGS and MDT0000, MDT0001, OSS, client.
- tcp and tcp999 on each server. tcp is the tenant network. tcp999 is the network for the targets.
Analysis
With the two checks removed, the option alone does not keep traffic on the net. There are three gaps:
- ptl_send_rpc() gives LNet no source NID. With a Multi-Rail peer, LNet picks any local NI and any peer NI.
- ptlrpc_connection_get() replaces the peer NID with the primary NID, which can be on another net. A source NID alone does not help. LNet treats a destination NID on another net as routed, and the send fails.
- Server-initiated requests use the reverse import, which has no restricted net. In a test run, the MGS sent a blocking AST to an MDS on tcp, with the targets mounted network=tcp999.
Proposed fix
- Do not delete a peer that Lustre added when a ping reply shows discovery off.
- When an import selects a connection, find a source NID and a peer NID on the restricted net. ptl_send_rpc() gives both NIDs to LNet.
- When a peer connects to a target on the restricted net of the target, give the reverse import the NIDs of the connect request.
- Remove the check in server_get_mti(). Apply the check in lmd_parse() to client mounts only.
- Add a test that counts the PUT messages between servers on each net.
- is related to
-
LU-21288 lnet: discovery-off push deletes a configured peer
-
- Open
-