https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14803 Lustre <p>IB access flags "IB_ACCESS_RELAXED_ORDERING/IBV_ACCESS_RELAXED_ORDERING" was added in 5.6 (commit 2233c6609c11146ed1a26eec2e4335131077a608: "RDMA/uverbs: Add new relaxed ordering memory region access flag") and in MOFED-5.0-1.</p> <p>Since MOFED 5.1-0.6.6.0, "relaxed ordering" PCI feature is supported for ConnectX-4 and above:</p> <blockquote><p>Relaxed ordering is a PCIe feature which allows flexibility in the transaction order over the PCIe. This reduces the number of retransmissions on the lane, and increases performance up to 4 times.<br/> <span class="error">&#91;...&#93;</span></p></blockquote> <p>ref: <a href="https://docs.mellanox.com/display/MLNXOFEDv531050/Release+Notes+Change+Log+History" class="external-link" target="_blank" rel="nofollow noopener">https://docs.mellanox.com/display/MLNXOFEDv531050/Release+Notes+Change+Log+History</a></p> <p>The following patch add the flag IB_ACCESS_RELAXED_ORDERING to ib access flags and add ko2iblnd parameter "ib_relaxed_ordering" to deactivate/activate the feature (if the driver or kernel have buggy implementation).</p> LU-14803

Activate relaxed ordering optimization on MOFED cards

Improvement Minor Open Unresolved Etienne Aujames Etienne Aujames Thu, 1 Jul 2021 13:14:00 +0000 Fri, 8 Oct 2021 17:34:48 +0000 0 6 <p>Etienne AUJAMES (eaujames@ddn.com) uploaded a new patch: <a href="https://review.whamcloud.com/44125" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44125</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14803" title="Activate relaxed ordering optimization on MOFED cards" class="issue-link" data-issue-key="LU-14803">LU-14803</a> o2iblnd: Activate relaxed ordering optimization<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cc3747a2678a0ebde553a5e78b1563943c20f258</p> <p>Hello Etienne,</p> <p>It is useful from PCie <b>Gen4</b> Mellanox card ie &gt;= ConnectX-6. We have seen huge improvement (with large size lnet message) +40% in bandwidth (We forced "relaxed ordering" in burning mlx card config, because we had Mofed 4.7).<br/> One question, the feature seems available from MOFED 5.1-0.6.6.0 but not from MOFED 5.0.1 (as you have mentioned in your description), can you confirm it?</p> <p>Gael</p> <p> </p> <p>The patch above does not work with MOFED5.1 (IB_ACCESS_RELAXED_ORDERING flag is not supported for Fast Memory Region):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [Mon Sep 27 10:36:55 2021] LNet: Added LNI 172.18.4.1@o2ib [32/1024/0/180] [Mon Sep 27 10:37:01 2021] infiniband mlx5_0: set_reg_wr:839:(pid 17412): Fast update of atomic access <span class="code-keyword">for</span> MR is disabled [Mon Sep 27 10:37:01 2021] LNetError: 17412:0:(o2iblnd_cb.c:1031:kiblnd_post_tx_locked()) Error -22 posting transmit to 172.18.4.2@o2ib [Mon Sep 27 10:37:01 2021] infiniband mlx5_0: set_reg_wr:839:(pid 17410): Fast update of atomic access <span class="code-keyword">for</span> MR is disabled [Mon Sep 27 10:37:01 2021] infiniband mlx5_0: set_reg_wr:839:(pid 17409): Fast update of atomic access <span class="code-keyword">for</span> MR is disabled </pre> </div></div> <p>After reading some MOFED commit, it seems that MOFED 5.2 support the flag but only for ConnectX-7 cards:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> commit 896ec9735336f5adb576d372ed7e411bce2fc74c Author: Meir Lichtinger &lt;meirl@mellanox.com&gt; Date: Thu Jul 16 13:52:48 2020 +0300 RDMA/mlx5: Set mkey relaxed ordering by UMR with ConnectX-7 Up to ConnectX-7 UMR is not used when user passes relaxed ordering access flag. ConnectX-7 supports setting relaxed ordering read/write mkey attribute by UMR, indicated by <span class="code-keyword">new</span> HCA capabilities. With ConnectX-7 driver uses UMR when user set relaxed ordering access flag, in contrast to previous silicon models. Specifically it includes setting relvant flags of mkey context mask in UMR control segment, and relaxed ordering write and read flags in UMR mkey context segment. </pre> </div></div> <p>On MOFED 5.4 relaxed ordering is activated by default for all kernel services. For this purpose flag IB_ACCESS_RELAXED_ORDERING is changed to IB_ACCESS_DISABLE_RELAXED_ORDERING.<br/> <a href="https://patchwork.kernel.org/project/linux-rdma/cover/cover.1621505111.git.leonro@nvidia.com/" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.kernel.org/project/linux-rdma/cover/cover.1621505111.git.leonro@nvidia.com/</a></p> <p>So this patch <a href="https://review.whamcloud.com/44125" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44125</a> will work only for MOFED-5.2/5.3 for ConnectX-7 cards, which not really useful.</p> Development Rank 1|i01y9j: Rank (Obsolete) 9223372036854775807