[LU-8943] Enable Multiple IB/OPA Endpoints Between Nodes

• Type: Improvement
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0
• Labels: lnet
• Assignee: Doug Oucharek (Inactive)
• Reporter: Doug Oucharek (Inactive)

Description

OPA driver optimizations are based on the MPI model where it is expected to have multiple endpoints between two given nodes. To enable this optimization for Lustre, we need to make it possible, via an LND-specific tuneable, to create multiple endpoints and to balance the traffic over them.

I have already created an experimental patch to test this theory out. I was able to push OPA performance to 12.4GB/s by just having 2 QPs between the nodes and round robin messages between them.

This Jira ticket is for productizing my patch and testing it out thoroughly for OPA and IB. Test results will be posted to this ticket.

Issue Links

• has to be finished together with: LUDOC-374 Add notes about conns_per_peer ko2iblnd parameter