[LU-9472] FastReg (MLX5) support breaks when map_on_demand > 0

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.0, Upstream
• Affects Version/s: Lustre 2.10.0, Upstream
• Labels: lnet
• Severity: 3
• Assignee: Doug Oucharek (Inactive)
• Reporter: Doug Oucharek (Inactive)

Description

When building against MODED 4, the default for map_on_demand switches from 0 to 256.  This is breaking MLX5-based cards which make use of the FastReg support in ko2iblnd.  There are three problems with FastReg which need to be fixed:

1. In kiblnd_fmr_pool_map() when using elements from the fpo_pool_list, if the list runs out, the current code is setting rc to -EBUSY when it should be -EAGAIN.  EAGAIN triggers the pool to be made bigger.  EBUSY just fails the transfer and connection (not what we want).
2. Even after I fix the setting of rc in number 1, bringing down the network via "lctl network down" trips this assert: 
   [ 1172.255552] LNetError: 10176:0:(o2iblnd.c:1421:kiblnd_destroy_fmr_pool()) ASSERTION( fpo->fpo_map_count == 0 ) failed: 

1. Every time the pool size is increased, I keep seeing this annoying log (with neterror on): 
   May  9 00:22:26 trevis-407 kernel: LNet: Using FastReg for registration

The first 2 items are blockers and must be fixed ASAP.  The 3rd might as well be addressed at the same time.

 

Issue Links

• is duplicated by: LU-9932 LU-9026, LU-9500, and LU-9472 backports
• is related to: LU-9500 MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe
• is related to: LU-6215 Sync Lustre external tree with lustre linux kernel client

Comments

[James A Simmons]

Not yet.

[Doug Oucharek (Inactive)]

Has this been pushed upstream yet?

[Peter Jones]

Landed for 2.10

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/27015/
Subject: LU-9472 lnd: Fix FastReg map/unmap for MLX5
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: b436c75d9488222190de8b30f56d720f8ec63d6f

[Alexey Lyashkov]

Tested fix and it work for me.

[Gerrit Updater]

Doug Oucharek ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27015
Subject: LU-9472 lnd: Fix FastReg map/unmap for MLX5
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a6a1d45a72360b5cc7e9e3a65c7456fa62c19192

[James A Simmons]

Thanks for finding this.