[LU-14803] Activate relaxed ordering optimization on MOFED cards

• Status: Open
• Project: Lustre
• Type: Improvement
• Priority: Minor
• Reporter: Etienne Aujames
• Assignee: Etienne Aujames
• Resolution: Unresolved

Description

IB access flags "IB_ACCESS_RELAXED_ORDERING/IBV_ACCESS_RELAXED_ORDERING" was added in 5.6 (commit 2233c6609c11146ed1a26eec2e4335131077a608: "RDMA/uverbs: Add new relaxed ordering memory region access flag") and in MOFED-5.0-1.

Since MOFED 5.1-0.6.6.0, "relaxed ordering" PCI feature is supported for ConnectX-4 and above:

Relaxed ordering is a PCIe feature which allows flexibility in the transaction order over the PCIe. This reduces the number of retransmissions on the lane, and increases performance up to 4 times.
[...]

ref: https://docs.mellanox.com/display/MLNXOFEDv531050/Release+Notes+Change+Log+History

The following patch add the flag IB_ACCESS_RELAXED_ORDERING to ib access flags and add ko2iblnd parameter "ib_relaxed_ordering" to deactivate/activate the feature (if the driver or kernel have buggy implementation).

Comments

[Gerrit Updater]

Etienne AUJAMES ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44125
Subject: LU-14803 o2iblnd: Activate relaxed ordering optimization
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cc3747a2678a0ebde553a5e78b1563943c20f258

[DELBARY Gael]

Hello Etienne,

It is useful from PCie Gen4 Mellanox card ie >= ConnectX-6. We have seen huge improvement (with large size lnet message) +40% in bandwidth (We forced "relaxed ordering" in burning mlx card config, because we had Mofed 4.7).
One question, the feature seems available from MOFED 5.1-0.6.6.0 but not from MOFED 5.0.1 (as you have mentioned in your description), can you confirm it?

Gael

 

[Etienne Aujames]

The patch above does not work with MOFED5.1 (IB_ACCESS_RELAXED_ORDERING flag is not supported for Fast Memory Region):

[Mon Sep 27 10:36:55 2021] LNet: Added LNI 172.18.4.1@o2ib [32/1024/0/180]
[Mon Sep 27 10:37:01 2021] infiniband mlx5_0: set_reg_wr:839:(pid 17412): Fast update of atomic access for MR is disabled
[Mon Sep 27 10:37:01 2021] LNetError: 17412:0:(o2iblnd_cb.c:1031:kiblnd_post_tx_locked()) Error -22 posting transmit to 172.18.4.2@o2ib
[Mon Sep 27 10:37:01 2021] infiniband mlx5_0: set_reg_wr:839:(pid 17410): Fast update of atomic access for MR is disabled
[Mon Sep 27 10:37:01 2021] infiniband mlx5_0: set_reg_wr:839:(pid 17409): Fast update of atomic access for MR is disabled

After reading some MOFED commit, it seems that MOFED 5.2 support the flag but only for ConnectX-7 cards:

    commit 896ec9735336f5adb576d372ed7e411bce2fc74c
    Author: Meir Lichtinger <[e-mail]>
    Date:   Thu Jul 16 13:52:48 2020 +0300

        RDMA/mlx5: Set mkey relaxed ordering by UMR with ConnectX-7
        
        Up to ConnectX-7 UMR is not used when user passes relaxed ordering access
        flag. ConnectX-7 supports setting relaxed ordering read/write mkey
        attribute by UMR, indicated by new HCA capabilities.
        
        With ConnectX-7 driver uses UMR when user set relaxed ordering access
        flag, in contrast to previous silicon models. Specifically it includes
        setting relvant flags of mkey context mask in UMR control segment, and
        relaxed ordering write and read flags in UMR mkey context segment.

On MOFED 5.4 relaxed ordering is activated by default for all kernel services. For this purpose flag IB_ACCESS_RELAXED_ORDERING is changed to IB_ACCESS_DISABLE_RELAXED_ORDERING.
https://patchwork.kernel.org/project/linux-rdma/cover/[e-mail]/

So this patch https://review.whamcloud.com/44125 will work only for MOFED-5.2/5.3 for ConnectX-7 cards, which not really useful.