Problem

After a QP flush, a FastReg descriptor can keep an MR key that the HCA does not know. Each later use of that descriptor fails its LOCAL_INV, and the QP goes to error. Each conn that gets the descriptor fails the same way. Bulk traffic stops until o2iblnd frees the idle pool. The idle check skips the first pool, so its descriptors stay bad.

Environment

• master at 4a7bddaf15. The code is from f2600449ce ("LU-5783 o2iblnd: Add Fast Reg memory registration support").
• o2ib with FastReg. We saw it on Soft-RoCE (rxe).

Reproduction

1. Set up 2 VMs with Soft-RoCE and LNet on o2ib.
2. Start lst brw with 1M reads and writes between the 2 nodes.
3. Drop the RoCE traffic for 5 s during the load. Repeat 6 times.
4. Look at the lst bulk rate and at the rxe_invalidate_mr() failures.

without the fix: bulk 0 MiB/s after cycle 1, 43 LOCAL_INV failures in rxe_invalidate_mr()
with the fix:    bulk 14-30 MiB/s in each cycle, no LOCAL_INV failures, 29 MR renewals

Analysis

o2iblnd posts LOCAL_INV and REG_MR before a FastReg RDMA. It bumps the MR key when it maps the descriptor. A QP flush can drop both work requests, so the HCA keeps the old key. The next use posts LOCAL_INV with the new key, and the HCA rejects it (rxe_invalidate_mr() returns -EINVAL, and the completion is IB_WC_LOC_QP_OP_ERR).

LU-16349 (0c93919f13) fixed only a descriptor that o2iblnd maps but never posts.

lnet_send_error_simulation() has the same effect. It posts nothing, but kiblnd_post_tx_locked() marks the LOCAL_INV as used.

Proposed fix

A tx can fail after o2iblnd posts its LOCAL_INV and REG_MR. Then kiblnd_fmr_pool_unmap() replaces the MR of the descriptor with a new one. If the allocation fails, free the descriptor. For a simulated send error, keep the LOCAL_INV for the next post.