[LU-9179] Upstream ko2iblnd has poor performance

Related — Sat, 3 Nov 2018 18:37:50 +0000

On a MLX QDR system, I get the following performance with current master on RHEL 7.3:

Read: 3.1 GB/s, Write: 3.15 GB/s

With the latest upstream build and ~~LU-9026~~ fix, I am getting:

Read: 1.25 GB/s, Write: 1.13 GB/s

To see if the problem is due to ~~LU-9026~~, I went back to before the RDMA API changes which broke ko2iblnd (4.8 rc2) and got:

Read: 0.63 GB/s, Write: 0.62 GB/s

So, I feel we have a bad problem with upstream LNet IB performance. It is possible that lnet-selftest is broken (certainly for 4.8rc2, that is possible).

I'm still unable to validate ~~LU-9026~~ on the upstream client. In theory, I get the same effect on master by setting map_on_demand to 256. When I do that, I see about a 5% drop in performance only. So, my suspicion is we have a problem with ko2i

Whamcloud Community JIRA

[LU-9179] Upstream ko2iblnd has poor performance