[LU-8943] Enable Multiple IB/OPA Endpoints Between Nodes

• Type: Improvement
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0
• Labels: lnet
• Assignee: Doug Oucharek (Inactive)
• Reporter: Doug Oucharek (Inactive)

Description

OPA driver optimizations are based on the MPI model where it is expected to have multiple endpoints between two given nodes. To enable this optimization for Lustre, we need to make it possible, via an LND-specific tuneable, to create multiple endpoints and to balance the traffic over them.

I have already created an experimental patch to test this theory out. I was able to push OPA performance to 12.4GB/s by just having 2 QPs between the nodes and round robin messages between them.

This Jira ticket is for productizing my patch and testing it out thoroughly for OPA and IB. Test results will be posted to this ticket.

Issue Links

• has to be finished together with: LUDOC-374 Add notes about conns_per_peer ko2iblnd parameter

Comments

[Doug Oucharek (Inactive)]

Dmitry, when you get the file system mounted, can you issue the following sequence on both nodes to ensure we are creating 4 connections on each:

 

lctl
> network o2ib
> conn_list

You should see 4 connections to the peer if the initiator (usually the client) has the MultiQP patch, and 1 connection to the peer if it doesn't.

[Doug Oucharek (Inactive)]

I just tried to reproduce with the passive node being unpatched.  Was not able to reproduce your issue.  The "lctl network down" takes a long time, but does succeed.  There must be something else here.  Do you know if your parameters like map_on_demand are different?  Is a reconnection happening to renegotiate the parameters?  This is something I have not tried.

[Doug Oucharek (Inactive)]

That might be the reason.  The client will create multiple connections, but the server will only have one they are all talking to.  When one connection on the client is closed, the connection on the server will be closed.  I suspect the remaining connections on the client can't be closed.  I'll have to look at the code to see what I can do in this situation.

I suspect if the server has the patch, you would not have a problem.

[Dmitry Eremin (Inactive)]

I'm using new Lustre client with this patch and old Lustre servers without this patch. So, I just mount lustre FS then use it and then try to unload after umount. I don't use DLC. I have CentOS 7.3 in both sites.

 

[Doug Oucharek (Inactive)]

When I created the performance spreadsheet, I needed to keep changing conns_per_peer.  I had no problems taking down and brining up LNet using these commands:

Up:

modprobe lnet
lctl network configure
modprobe lnet-selftest

Down:

rmmod lnet-selftest
lctl network down
rmmod ko2iblnd
rmmod lnet

There must be something different about what you are doing which is triggering ref counters to not be reduced. Are you using DLC? What is your environment?  Are both nodes running the latest code with this patch?

[Dmitry Eremin (Inactive)]

No, as I mentoined before only reboot helps.

# lustre_rmmod                                                                  
rmmod: ERROR: Module ko2iblnd is in use

# lsmod|less                                                                    
Module                  Size  Used by
ko2iblnd              233790  1 
ptlrpc               1343928  0 
obdclass             1744518  1 ptlrpc
lnet                  483843  3 ko2iblnd,obdclass,ptlrpc
libcfs                416336  4 lnet,ko2iblnd,obdclass,ptlrpc
[...]

# lctl network down                                                             
LNET busy

lnetctl > lnet unconfigure
unconfigure:
    - lnet:
          errno: -16
          descr: "LNet unconfigure error: Device or resource busy"
lnetctl > lnet unconfigure --all
unconfigure:
    - lnet:
          errno: -16
          descr: "LNet unconfigure error: Device or resource busy"

# lustre_rmmod                                                                  
rmmod: ERROR: Module ko2iblnd is in use

[Andreas Dilger]

Does "lctl network down" or "lnetctl lnet unconfigure" help?

[Dmitry Eremin (Inactive)]

I observed strange behavior. It looks after this commit I cannot unload ko2iblnd module. LNet is busy even all unmounted successfully. Only reboot helps.

 

 

[Peter Jones]

Landed for 2.10

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/25168/
Subject: LU-8943 lnd: Enable Multiple OPA Endpoints between Nodes
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7241e68f37962991ef43a6c01b3a83ff67282d88

[Doug Oucharek (Inactive)]

I did update the OPA defaults to set conns_per_peer to 4 when OPA is detected.  I'll also update the manual under LUDOC-374.

I bumped the conns_per_peer to 4 from 3 because OPA team is going to start recommending a krcvqs default of 4 especially for a low number of cores (i.e. VMs).  Having a conns_per_peer of 4 helps to compensate for the lower krcvqs number so we should work well out of the box whether krcvqs is 4 or 8.