MLX5 does not support FMR. With the removal of PMR (LU-6850) we need a way to support MLX5.
To do so the following solution is devised. FMR is enabled by setting map-on-demand to: 0 < value <=256.
This represents a problem for coexistence nodes with both OPA and MLX5. OPA performance is greatly enhanced by setting map-on-demand, however if map-on-demand is set for MLX5, then it will not work.
We need to be able to support per-NI map-on-demand value; therefore when OPA net is configured it can use optimal map-on-demand value, but when MLX5 is configured map-on-demand can be set to 0, disabling it.
However, this raises another issue which is different map-on-demand values across fabrics. LNet currently doesn't support this. However LU-3322 adds support for this scenario. This problem is relevant for both OPA support and for support of clusters which have both MLX5 and MLX4 interconnected.
The proposed solution is consistent of three patches.
1. LU-6850 patch which removed support for PMR
2. LU-3322 support for different map-on-demand and peertxcredits values across the network.
3. The patch for this LU, which adds support for per NI map-on-demand.
Future patch will add support for dynamic setting of map-on-demand, but that's a future feature not required to address the immediate need.
Bug
Major
