[LU-12815] Create multiple TCP sockets per SockLND

• Type: Technical task
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.15.0, Lustre 2.12.10
• Labels: performance
• Assignee: Amir Shehata (Inactive)
• Reporter: Andreas Dilger

Description

For high-bandwidth Ethernet interfaces (e.g. 100GigE), it would be useful to create multiple TCP connections per interface for bulk transfers in order to maximize performance (i.e. conns_per_peer=4 for socklnd in addition to o2iblnd). We already have three separate TCP connections per LND - read, write, and small message.

For large clusters this may be problematic because of the number of TCP connections to a server, but for smaller configurations this could be very useful.

Issue Links

• is duplicated by: LU-14293 Poor lnet/ksocklnd(?) performance on 2x100G bonded ethernet
• is related to: LU-14676 Better hash distribution to different CPTs when LNET router is exist

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/47252/
Subject: LU-12815 socklnd: add conns_per_peer parameter
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: be2c4bb928b5cf6b428d7974e8fd89ea177fa2df

[Gerrit Updater]

"Cyril Bordage <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/47252
Subject: LU-12815 socklnd: add conns_per_peer parameter
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: b370ffba7b778f9b5fee325a8c67228ca2454137

[Peter Jones]

Looks like everything has landed for 2.15

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44417/
Subject: LU-12815 socklnd: set conns_per_peer based on link speed
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: c44afcfb72a1c2fd8392bfab3143c3835b146be6

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/41463/
Subject: LU-12815 socklnd: allow dynamic setting of conns_per_peer
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: a5cbe7883db6d77b82fbd83ad4c662499421d229

[Gerrit Updater]

Serguei Smirnov ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44417
Subject: LU-12815 socklnd: set conns_per_peer based on link speed
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: ca2f4fed6d85d2e4506958fcc2e1c6c98eb2d020

[Andreas Dilger]

There may still be work needed to distribute RPCs from a single client to multiple CPTs on the server, in order to get the best performance for real IO workloads.

Otherwise, a client with a single interface (NID) will have all of its RPCs handled by cores in a single CPT, which is not quite the same as having multiple real interfaces on the client. Discussion is ongoing in LU-14676.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/41056/
Subject: LU-12815 socklnd: add conns_per_peer parameter
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 71b2476e4ddb95aa42f4a0ea3f23b1826017bfa5

[Andreas Dilger]

Does multiple TCP sockets featurer will only apply to Bulk I/O sockets?

Correct, this is only for bulk sockets.

[Aurelien Degremont (Inactive)]

That's correct. I wanted to warn about this often unknown use case (I was really surprised when I discovered it), with any potential impact of this feature.

Agreed that some sites could prefer risking evictions than changing firewall rules. However, this is surprising behavior and sites tends to reduce as much as possible evictions and having a "normal case" where evictions will happen but are expected, just because of unknown traffic rules is not the best option. This could cause additional JIRA tickets. But this problem already exists and is independent of this ticket. I just wanted to avoid increasing the issue. Does multiple TCP sockets featurer will only apply to Bulk I/O sockets?