[LU-17513] how does 'conns_per_peer' apply with multiple NIDs?

Related — Thu, 8 Feb 2024 00:05:12 +0000

On a system with multiple NIDs on both peers in a connection, there is an issue of creating conns_per_peer connections to each of the NIDs on each local/remote NID combination. AFAIK, that means a single pair of nodes with conns_per_peer=4 and 4 interfaces on each side (e.g. DGX + server) would create (4 conns * <read,write> + 1 control) * 2 local NIDs * 2 remote NIDs = 36 TCP sockets PER PEER, and this increases with the square of the number of NIDs on each peer (=144 sockets PER PEER with 4 NIDs each).

With 1200 2-NID clients that means 43200 sockets, which is a real-world problem today, and could make a slightly larger Ethernet cluster unusable due to connection storms exceeding the total number of available ports on the server.

It seems likely (although this is speculation on my part) that socklnd should distribute conns_per_peer across the available NID combinations, and then send messages to the peer with the knowledge of which peer NIDs actually have established connections instead of establishing an all-to-all set of connections. In most cases, the physical network topology (switches) would likely prefer a 1:1 grouping of NIDs from the local and remote interfaces. In the case of interface failure on the local/remote NID then those connections could be redistributed to other local or remote NIDs if needed.

Whamcloud Community JIRA

[LU-17513] how does 'conns_per_peer' apply with multiple NIDs?