Details
-
Improvement
-
Resolution: Fixed
-
Critical
-
None
-
9223372036854775807
Description
OPA driver optimizations are based on the MPI model where it is expected to have multiple endpoints between two given nodes. To enable this optimization for Lustre, we need to make it possible, via an LND-specific tuneable, to create multiple endpoints and to balance the traffic over them.
I have already created an experimental patch to test this theory out. I was able to push OPA performance to 12.4GB/s by just having 2 QPs between the nodes and round robin messages between them.
This Jira ticket is for productizing my patch and testing it out thoroughly for OPA and IB. Test results will be posted to this ticket.
Attachments
Issue Links
- has to be finished together with
-
-
- Resolved
-
Activity
| Link | New: This issue is related to CRAY-6 [ CRAY-6 ] |
| Link | New: This issue is related to LDEV-301 [ LDEV-301 ] |
| Link | New: This issue is related to DDN-453 [ DDN-453 ] |
| Link | New: This issue is related to JFC-17 [ JFC-17 ] |
| Link | New: This issue is duplicated by HUAW-46 [ HUAW-46 ] |
Would it be a good idea to track all this under a new ticket instead of tacking onto an already closed one?
Cliff is seeing this same problem on the soak cluster but there is no OPA, only MLX IB. I'm beginning to wonder if this is a problem with the Mutli-Rail drop rather than this change.
From server:
# lctl lctl > network o2ib lctl > conn_list 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 192.168.213.125@o2ib mtu -1 ...
Cliff created a ticket for this already:
LU-9524.Summary: this appears to have been introduced in patch: https://review.whamcloud.com/#/c/26959/ and not the change under this ticket. ptlrpc is not longer being unloaded with lustre_rmmod so lnet won't unload.