On a MLX QDR system, I get the following performance with current master on RHEL 7.3:

Read: 3.1 GB/s, Write: 3.15 GB/s

With the latest upstream build and LU-9026 fix, I am getting:

Read: 1.25 GB/s, Write: 1.13 GB/s

To see if the problem is due to LU-9026, I went back to before the RDMA API changes which broke ko2iblnd (4.8 rc2) and got:

Read: 0.63 GB/s, Write: 0.62 GB/s

So, I feel we have a bad problem with upstream LNet IB performance.  It is possible that lnet-selftest is broken (certainly for 4.8rc2, that is possible).

I'm still unable to validate LU-9026 on the upstream client.  In theory, I get the same effect on master by setting map_on_demand to 256.  When I do that, I see about a 5% drop in performance only.  So, my suspicion is we have a problem with ko2i