[LU-8943] Enable Multiple IB/OPA Endpoints Between Nodes

• Type: Improvement
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0
• Labels: lnet
• Assignee: Doug Oucharek (Inactive)
• Reporter: Doug Oucharek (Inactive)

Description

OPA driver optimizations are based on the MPI model where it is expected to have multiple endpoints between two given nodes. To enable this optimization for Lustre, we need to make it possible, via an LND-specific tuneable, to create multiple endpoints and to balance the traffic over them.

I have already created an experimental patch to test this theory out. I was able to push OPA performance to 12.4GB/s by just having 2 QPs between the nodes and round robin messages between them.

This Jira ticket is for productizing my patch and testing it out thoroughly for OPA and IB. Test results will be posted to this ticket.

Issue Links

• has to be finished together with: LUDOC-374 Add notes about conns_per_peer ko2iblnd parameter

Comments

[Doug Oucharek (Inactive)]

Cliff created a ticket for this already: LU-9524.

Summary: this appears to have been introduced in patch: https://review.whamcloud.com/#/c/26959/ and not the change under this ticket. ptlrpc is not longer being unloaded with lustre_rmmod so lnet won't unload.

[Peter Jones]

Would it be a good idea to track all this under a new ticket instead of tacking onto an already closed one?

[Doug Oucharek (Inactive)]

Cliff is seeing this same problem on the soak cluster but there is no OPA, only MLX IB. I'm beginning to wonder if this is a problem with the Mutli-Rail drop rather than this change.

[Dmitry Eremin (Inactive)]

From server:

# lctl                                                                          
lctl > network o2ib
lctl > conn_list
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
...

[Dmitry Eremin (Inactive)]

192.168.213.125@o2ib - client

[Dmitry Eremin (Inactive)]

# lctl
lctl > network o2ib
lctl > conn_list
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.125@o2ib mtu -1
192.168.213.231@o2ib mtu -1
192.168.213.231@o2ib mtu -1
192.168.213.231@o2ib mtu -1
192.168.213.231@o2ib mtu -1
192.168.213.232@o2ib mtu -1
192.168.213.232@o2ib mtu -1
192.168.213.232@o2ib mtu -1
192.168.213.232@o2ib mtu -1
192.168.213.233@o2ib mtu -1
192.168.213.233@o2ib mtu -1
192.168.213.233@o2ib mtu -1
192.168.213.233@o2ib mtu -1
192.168.213.234@o2ib mtu -1
192.168.213.234@o2ib mtu -1
192.168.213.234@o2ib mtu -1
192.168.213.234@o2ib mtu -1
192.168.213.235@o2ib mtu -1
192.168.213.235@o2ib mtu -1
192.168.213.235@o2ib mtu -1
192.168.213.235@o2ib mtu -1
192.168.213.236@o2ib mtu -1
192.168.213.236@o2ib mtu -1
192.168.213.236@o2ib mtu -1
192.168.213.236@o2ib mtu -1

# lnetctl lnet unconfigure --all                                                
unconfigure:
    - lnet:
          errno: -16
          descr: "LNet unconfigure error: Device or resource busy"

Client: 2.9.57_48_g0386263
Servers:
lustre: 2.7.19.10
kernel: patchless_client
build: 2.7.19.10--PRISTINE-3.10.0-514.10.2.el7_lustre.x86_64

[Doug Oucharek (Inactive)]

Dmitry, when you get the file system mounted, can you issue the following sequence on both nodes to ensure we are creating 4 connections on each:

 

lctl
> network o2ib
> conn_list

You should see 4 connections to the peer if the initiator (usually the client) has the MultiQP patch, and 1 connection to the peer if it doesn't.

[Doug Oucharek (Inactive)]

I just tried to reproduce with the passive node being unpatched.  Was not able to reproduce your issue.  The "lctl network down" takes a long time, but does succeed.  There must be something else here.  Do you know if your parameters like map_on_demand are different?  Is a reconnection happening to renegotiate the parameters?  This is something I have not tried.

[Doug Oucharek (Inactive)]

That might be the reason.  The client will create multiple connections, but the server will only have one they are all talking to.  When one connection on the client is closed, the connection on the server will be closed.  I suspect the remaining connections on the client can't be closed.  I'll have to look at the code to see what I can do in this situation.

I suspect if the server has the patch, you would not have a problem.

[Dmitry Eremin (Inactive)]

I'm using new Lustre client with this patch and old Lustre servers without this patch. So, I just mount lustre FS then use it and then try to unload after umount. I don't use DLC. I have CentOS 7.3 in both sites.

 

[Doug Oucharek (Inactive)]

When I created the performance spreadsheet, I needed to keep changing conns_per_peer.  I had no problems taking down and brining up LNet using these commands:

Up:

modprobe lnet
lctl network configure
modprobe lnet-selftest

Down:

rmmod lnet-selftest
lctl network down
rmmod ko2iblnd
rmmod lnet

There must be something different about what you are doing which is triggering ref counters to not be reduced. Are you using DLC? What is your environment?  Are both nodes running the latest code with this patch?