[LU-9500] MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.1, Lustre 2.11.0
• Severity: 3
• Assignee: Sonia Sharma (Inactive)
• Reporter: Doug Oucharek (Inactive)

Description

In Lustre, we allow the first fragment in an IOV-based message to be non-page aligned.  When we set up the scatter/gather list we properly set the address and page_offset to reflect the non-page alignment.  

When we assign a remote address for RDMA purposes, the current code masks the address so it is page aligned.  When the page aligned address does not match the address in the scatter/gather list, the mlx5 driver under MOFED 4 is rejecting the IB_RDMA_WRITE operation by doing a "dump_cqe" error message.

That is the main problem to be fixed.  However, the code which was doing the masking for page alignment is wrong.  Here is the line of code in the routine kiblnd_fmr_map_tx() which is doing the masking incorrectly:

rd->rd_frags[0].rf_addr &= ~hdev->ibh_page_mask;

The "~" should not be there. We were setting the rf_addr to the page offset. When pages are aligned, rf_addr becomes zero and that is the remote_addr value we send to the other node. The fact that this works and does not break things sort of implies that the MOFED code is not using the remote_addr field of a IB_RDMA_WRITE work request.

In any case, we need to fix this in case some day some code does actually pay attention to this field.

The question to be answered here: should the remote address we generate be page aligned or not. When I stopped page aligning it, the dump_cqe error stopped and everything worked just fine.
 

Issue Links

• is blocked by: LU-9565 lnet-selftest: Newly added "off" parameter no longer working
• is duplicated by: LU-9932 LU-9026, LU-9500, and LU-9472 backports
• is related to: LU-9472 FastReg (MLX5) support breaks when map_on_demand > 0
• is related to: LU-9983 LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS
• is related to: LU-9461 lustre client mount fail after update IB driver and Lustre patch.
• is related to: LU-9679 Prepare lustre for adoption into the linux kernel