[LU-9472] FastReg (MLX5) support breaks when map_on_demand > 0

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0, Upstream
• Affects Version/s: Lustre 2.10.0, Upstream
• Labels: lnet
• Severity: 3
• Assignee: Doug Oucharek (Inactive)
• Reporter: Doug Oucharek (Inactive)

Description

When building against MODED 4, the default for map_on_demand switches from 0 to 256.  This is breaking MLX5-based cards which make use of the FastReg support in ko2iblnd.  There are three problems with FastReg which need to be fixed:

1. In kiblnd_fmr_pool_map() when using elements from the fpo_pool_list, if the list runs out, the current code is setting rc to -EBUSY when it should be -EAGAIN.  EAGAIN triggers the pool to be made bigger.  EBUSY just fails the transfer and connection (not what we want).
2. Even after I fix the setting of rc in number 1, bringing down the network via "lctl network down" trips this assert: 
   [ 1172.255552] LNetError: 10176:0:(o2iblnd.c:1421:kiblnd_destroy_fmr_pool()) ASSERTION( fpo->fpo_map_count == 0 ) failed: 

1. Every time the pool size is increased, I keep seeing this annoying log (with neterror on): 
   May  9 00:22:26 trevis-407 kernel: LNet: Using FastReg for registration

The first 2 items are blockers and must be fixed ASAP.  The 3rd might as well be addressed at the same time.

 

Issue Links

• is duplicated by: LU-9932 LU-9026, LU-9500, and LU-9472 backports
• is related to: LU-9500 MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe
• is related to: LU-6215 Sync Lustre external tree with lustre linux kernel client

Comments

[James A Simmons]

Not yet.

[Doug Oucharek (Inactive)]

Has this been pushed upstream yet?

[Peter Jones]

Landed for 2.10

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27015/
Subject: LU-9472 lnd: Fix FastReg map/unmap for MLX5
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b436c75d9488222190de8b30f56d720f8ec63d6f

[Alexey Lyashkov]

Tested fix and it work for me.

[Gerrit Updater]

Doug Oucharek ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27015
Subject: LU-9472 lnd: Fix FastReg map/unmap for MLX5
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a6a1d45a72360b5cc7e9e3a65c7456fa62c19192

[James A Simmons]

Thanks for finding this.

[Doug Oucharek (Inactive)]

The main problem, it turns out, is that the unmap routine is never being called for FastReg.  As such, we have to keep growing the pool and assert when trying to shut down networking (because pool items are leaking).