Details
-
Improvement
-
Resolution: Fixed
-
Critical
-
None
-
9223372036854775807
Description
OPA driver optimizations are based on the MPI model where it is expected to have multiple endpoints between two given nodes. To enable this optimization for Lustre, we need to make it possible, via an LND-specific tuneable, to create multiple endpoints and to balance the traffic over them.
I have already created an experimental patch to test this theory out. I was able to push OPA performance to 12.4GB/s by just having 2 QPs between the nodes and round robin messages between them.
This Jira ticket is for productizing my patch and testing it out thoroughly for OPA and IB. Test results will be posted to this ticket.
Attachments
Issue Links
- has to be finished together with
-
-
- Resolved
-
Activity
I'm using new Lustre client with this patch and old Lustre servers without this patch. So, I just mount lustre FS then use it and then try to unload after umount. I don't use DLC. I have CentOS 7.3 in both sites.
When I created the performance spreadsheet, I needed to keep changing conns_per_peer. I had no problems taking down and brining up LNet using these commands:
Up:
modprobe lnet lctl network configure modprobe lnet-selftest
Down:
rmmod lnet-selftest lctl network down rmmod ko2iblnd rmmod lnet
There must be something different about what you are doing which is triggering ref counters to not be reduced. Are you using DLC? What is your environment? Are both nodes running the latest code with this patch?
No, as I mentoined before only reboot helps.
# lustre_rmmod
rmmod: ERROR: Module ko2iblnd is in use
# lsmod|less
Module Size Used by
ko2iblnd 233790 1
ptlrpc 1343928 0
obdclass 1744518 1 ptlrpc
lnet 483843 3 ko2iblnd,obdclass,ptlrpc
libcfs 416336 4 lnet,ko2iblnd,obdclass,ptlrpc
[...]
# lctl network down
LNET busy
lnetctl > lnet unconfigure
unconfigure:
- lnet:
errno: -16
descr: "LNet unconfigure error: Device or resource busy"
lnetctl > lnet unconfigure --all
unconfigure:
- lnet:
errno: -16
descr: "LNet unconfigure error: Device or resource busy"
# lustre_rmmod
rmmod: ERROR: Module ko2iblnd is in use
Does "lctl network down" or "lnetctl lnet unconfigure" help?
I observed strange behavior. It looks after this commit I cannot unload ko2iblnd module. LNet is busy even all unmounted successfully. Only reboot helps.
Oleg Drokin (oleg.drokin@intel.com) merged in patch https://review.whamcloud.com/25168/
Subject: LU-8943 lnd: Enable Multiple OPA Endpoints between Nodes
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 7241e68f37962991ef43a6c01b3a83ff67282d88
I did update the OPA defaults to set conns_per_peer to 4 when OPA is detected. I'll also update the manual under LUDOC-374.
I bumped the conns_per_peer to 4 from 3 because OPA team is going to start recommending a krcvqs default of 4 especially for a low number of cores (i.e. VMs). Having a conns_per_peer of 4 helps to compensate for the lower krcvqs number so we should work well out of the box whether krcvqs is 4 or 8.
I recommend OPA systems with many cores use conns_per_peer = 3 and these HFI1 parameters:
options hfi1 krcvqs=8 piothreshold=0 sge_copy_mode=2 wss_threshold=70
Are you going to add these to the /usr/sbin/ko2iblnd-probe script, or be set by default in some other manner, or will this be up to the user to discover and set? At a very minimum there should be an update to the Lustre User Manual (see https://wiki.hpdd.intel.com/display/PUB/Making+changes+to+the+Lustre+Manual), but providing good performance out of the box is preferred.
Backwards compatibility testing looks good. An upgraded node who initiates connections will create conns_per_peer connections and the non-upgraded receiver node will allow that many connections to be created. However, the non-upgraded node will not "use" all the connections to send messages, only the first one. So performance will not improve.
If things are reversed (non-upgraded initiator to upgraded receiver) will work as if neither side is upgraded because it is the initiator who decides how many connections to have and in this case, it will just be one.
So, to get the performance benefit, both sides of a connection need to be upgraded with this patch and the initiator needs to have conns_per_peer set > 1.
Based on the attached spreadsheet, I recommend OPA systems with many cores use conns_per_peer = 3 and these HFI1 parameters:
options hfi1 krcvqs=8 piothreshold=0 sge_copy_mode=2 wss_threshold=70
However, if you are on a VM or have a limited number of cores, change conns_per_peer = 4 and krcvqs = 4 in the HFI1 parameters.
That might be the reason. The client will create multiple connections, but the server will only have one they are all talking to. When one connection on the client is closed, the connection on the server will be closed. I suspect the remaining connections on the client can't be closed. I'll have to look at the code to see what I can do in this situation.
I suspect if the server has the patch, you would not have a problem.