[LU-8943] Enable Multiple IB/OPA Endpoints Between Nodes

• Type: Improvement
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0
• Labels: lnet
• Assignee: Doug Oucharek (Inactive)
• Reporter: Doug Oucharek (Inactive)

Description

OPA driver optimizations are based on the MPI model where it is expected to have multiple endpoints between two given nodes. To enable this optimization for Lustre, we need to make it possible, via an LND-specific tuneable, to create multiple endpoints and to balance the traffic over them.

I have already created an experimental patch to test this theory out. I was able to push OPA performance to 12.4GB/s by just having 2 QPs between the nodes and round robin messages between them.

This Jira ticket is for productizing my patch and testing it out thoroughly for OPA and IB. Test results will be posted to this ticket.

Issue Links

• has to be finished together with: LUDOC-374 Add notes about conns_per_peer ko2iblnd parameter

Comments

[Doug Oucharek (Inactive)]

Cliff created a ticket for this already: LU-9524.

Summary: this appears to have been introduced in patch: https://review.whamcloud.com/#/c/26959/ and not the change under this ticket. ptlrpc is not longer being unloaded with lustre_rmmod so lnet won't unload.

[Peter Jones]

Would it be a good idea to track all this under a new ticket instead of tacking onto an already closed one?

[Doug Oucharek (Inactive)]

Cliff is seeing this same problem on the soak cluster but there is no OPA, only MLX IB. I'm beginning to wonder if this is a problem with the Mutli-Rail drop rather than this change.

[Dmitry Eremin (Inactive)]

From server:

# lctl                                                                          
lctl > network o2ib
lctl > conn_list
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
...

[Dmitry Eremin (Inactive)]

192.168.213.125@o2ib - client

[Dmitry Eremin (Inactive)]

# lctl
lctl > network o2ib
lctl > conn_list
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.231@o2ib mtu -1
192.168.213.231@o2ib mtu -1
192.168.213.231@o2ib mtu -1
192.168.213.231@o2ib mtu -1
192.168.213.232@o2ib mtu -1
192.168.213.232@o2ib mtu -1
192.168.213.232@o2ib mtu -1
192.168.213.232@o2ib mtu -1
192.168.213.233@o2ib mtu -1
192.168.213.233@o2ib mtu -1
192.168.213.233@o2ib mtu -1
192.168.213.233@o2ib mtu -1
192.168.213.234@o2ib mtu -1
192.168.213.234@o2ib mtu -1
192.168.213.234@o2ib mtu -1
192.168.213.234@o2ib mtu -1
192.168.213.235@o2ib mtu -1
192.168.213.235@o2ib mtu -1
192.168.213.235@o2ib mtu -1
192.168.213.235@o2ib mtu -1
192.168.213.236@o2ib mtu -1
192.168.213.236@o2ib mtu -1
192.168.213.236@o2ib mtu -1
192.168.213.236@o2ib mtu -1

# lnetctl lnet unconfigure --all                                                
unconfigure:
    - lnet:
          errno: -16
          descr: "LNet unconfigure error: Device or resource busy"

Client: 2.9.57_48_g0386263
Servers:
lustre: 2.7.19.10
kernel: patchless_client
build: 2.7.19.10--PRISTINE-3.10.0-514.10.2.el7_lustre.x86_64

[Doug Oucharek (Inactive)]

Dmitry, when you get the file system mounted, can you issue the following sequence on both nodes to ensure we are creating 4 connections on each:

 

lctl
> network o2ib
> conn_list

You should see 4 connections to the peer if the initiator (usually the client) has the MultiQP patch, and 1 connection to the peer if it doesn't.

[Doug Oucharek (Inactive)]

I just tried to reproduce with the passive node being unpatched.  Was not able to reproduce your issue.  The "lctl network down" takes a long time, but does succeed.  There must be something else here.  Do you know if your parameters like map_on_demand are different?  Is a reconnection happening to renegotiate the parameters?  This is something I have not tried.