https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17072 Lustre <p>During testing with master (2.15.57_130_g40c4041 / 40c404129b8ee51af5da7ec422672cc1eba74cbe) on EL9.2 with RoCE network, we noticed that LNet must have some dependencies with RoCE v1 being enabled. If only RoCE v2 is enabled and NOT v1, while the IB layer seems to work well (ib_write_bw, ibv_rc_pingpong, etc.), LNet doesn't work. Attaching a debug trace of a lctl ping on itself (using @o2ib), which doesn't succeed.</p> <p>In our case, enabling RoCE v1 on the hardware fixes the issue with LNet:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># bnxtnvm -dev=$ROCEIF setoption=disable_roce_v1#0 </pre> </div></div> Rocky Linux 9.2 – 5.14.0-284.25.1.el9_2.x86_64 - Broadcom BCM57414 LU-17072

LNet dependency to RoCE v1

Bug Minor Open Unresolved WC Triage Stephane Thiell Thu, 31 Aug 2023 21:43:23 +0000 Mon, 27 Nov 2023 20:02:56 +0000 Lustre 2.16.0 0 1 <p>Just to follow-up on this issue I reported: even after applying the workaround above, we had other LNet issues when using RoCE with Lustre on Broadcom NICs and realized that their RoCE support is actually not fully implemented with 57414 NICs. We switched to NVIDIA ConnectX-6 Lx NICs instead and RoCE seems to just work out of the box with Lustre.</p> Development Rank 1|i03udr: Rank (Obsolete) 9223372036854775807 Severity