Details
-
Improvement
-
Resolution: Cannot Reproduce
-
Major
-
None
-
None
-
None
-
5648
Description
Warpmech got LNet performance issue over 40GigE:
- between two 40GigE nodes, selftest can only get half of bandwidth
- it's not a big surprise to me, because socklnd connection is bound with exact one thread, and receiving side doesn't have Zero-copy, which means only one core is receiving from a 40GigE link if there is only one peer, so I suspect performance of 1:1 40GigE is CPU bound.
- one possible solution is reusing CONTROL link to transfer bulk data as well with some special policy
- I know that enable kiov vmap can help on receiving performance of some NICs, but not sure it will work for this case.
- while running BRW tests between 10GigE clients and 40GigE server, one direction can work well (saturate the bandwidth), another direction can only get half.
- 8 clients read can saturate the link pretty well, but I don't know if 4 clients can do the same.
- 8 clients write can't saturate the link, it can get half of bandwidth. I actually found that one 10GigE client can only get 250-300MB/sec on write against 40GigE server, which matches aggregation write performance of 8 clients (2GB/sec)
- network latency is quite low (I don't remember the number)