[LU-9179] Upstream ko2iblnd has poor performance

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Labels: lnet
• Severity: 3
• Assignee: WC Triage
• Reporter: Doug Oucharek (Inactive)

Description

On a MLX QDR system, I get the following performance with current master on RHEL 7.3:

Read: 3.1 GB/s, Write: 3.15 GB/s

With the latest upstream build and LU-9026 fix, I am getting:

Read: 1.25 GB/s, Write: 1.13 GB/s

To see if the problem is due to LU-9026, I went back to before the RDMA API changes which broke ko2iblnd (4.8 rc2) and got:

Read: 0.63 GB/s, Write: 0.62 GB/s

So, I feel we have a bad problem with upstream LNet IB performance.  It is possible that lnet-selftest is broken (certainly for 4.8rc2, that is possible).

I'm still unable to validate LU-9026 on the upstream client.  In theory, I get the same effect on master by setting map_on_demand to 256.  When I do that, I see about a 5% drop in performance only.  So, my suspicion is we have a problem with ko2i

Issue Links

• is related to: LU-9679 Prepare lustre for adoption into the linux kernel

Comments

[James A Simmons]

This was due to left overs from LU-7650 which was incorrect. All the code has been removed upstream and replaced with what landed during 2.11 development cycle.

[James A Simmons]

The LNet layer upstream is pretty much in sync with master just before multi-rail landed. The major difference is Al Viro's biovec patches are missing in master.