https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12901 Lustre <p>Attempting to bring up a file system in our test bed with the latest lustre version (2.13) I saw this new error on LNet bring up.</p> <p>[ 472.738363] LNet: 8481:0:(o2iblnd_cb.c:3395:kiblnd_check_conns()) Timed out tx for 10.37.248.232@o2ib1: 471 seconds<br/> [ 473.739295] LNetError: 2014:0:(o2iblnd.c:929:kiblnd_create_conn()) Can't create QP: -12, send_wr: 16317, recv_wr: 128, send_sge: 2, recv_sge: 1</p> <p>I found I can lower the peer_credits to get around this but that is not the proper fix.</p> <p> </p> <p> </p> <p> </p> <p> </p> Seen with newer Mellanox ConnectX-4 devices LU-12901

Failing to create a properly sized IB queue pair

Bug Major Resolved Fixed Serguei Smirnov James A Simmons Wed, 23 Oct 2019 18:15:32 +0000 Mon, 20 Dec 2021 21:03:35 +0000 Sun, 13 Dec 2020 15:44:59 +0000 Lustre 2.13.0 Lustre 2.14.0 1 11 <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a>,</p> <p>A few questions:</p> <p>Do you see it happen with MLNX 5?</p> <p>Do you see the issue occur immediately at start-up?</p> <p>Do you see the issue occur on a server with any number of client connections?</p> <p>Thanks,</p> <p>Serguei.</p> <p> </p> <p>Also seeing with Lustre 2.12.3 client on CentOS 7.6 (3.10.0-957.27.2.el7.x86_64). With both ConnectX-4 and -5 on client nodes.</p> <p>Server nodes are running Lustre 2.10.6 on CentOS 7.5 (3.10.0-862.14.4.el7.x86_64).  Servers are also mixed Mellanox EDR and Intel/Qlogic QDR.</p> <p>All nodes are using the in-box RDMA drivers.</p> <p>As most client nodes are still on Intel QDR, we previously set the ib0 peer_credits to 62 across all nodes. Lowering peer_credits to 42 on the problematic clients allows mounting of file system. </p> <p>Sorry we are also having issues with our IB switch so currently I'm not using our IB network.</p> <p>I also still see this on a Lustre client v2_12_3-98-g6db0c4f082 (+<a href="https://jira.whamcloud.com/browse/LU-12637" title="Support RHEL 8.1" class="issue-link" data-issue-key="LU-12637"><del>LU-12637</del></a> patch) on CentOS 8.1 (4.18.0-147.3.1.el8_1.x86_64) with ConnectX-4 using the CentOS RDMA drivers (mlx5_core) . I can also confirm that peer_credits=42 (instead of 63) works.</p> <p>I also see this on a Lustre client with Centos7.7 and ConnectX6 using Mellanox OFED 4.7</p> <p>I see this issue as well. CentOS 7.8, Lustre 2.13.0, MOFED 5.0-2.1.8, ConnectX6. Setting peer_credits to 128 fails as described. Lowering peer_credits to 48 results in functioning lnet.</p> <p> </p> <p> </p> <p>May I suggest to change the "<b>Affects Version/s</b>" attribute of this bug from 2.13.0 to 2.12.x (including 2.12.5 which is a LTS release). See e.g. the comments here or the reports on lustre-discuss.</p> <p>Serguei Smirnov (ssmirnov@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/40748" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40748</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12901" title="Failing to create a properly sized IB queue pair" class="issue-link" data-issue-key="LU-12901"><del>LU-12901</del></a> o2iblnd: retry qp creation with reduced queue depth<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: fe4fcd922196355b08981d9015f1635c88904fd3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40748/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40748/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12901" title="Failing to create a properly sized IB queue pair" class="issue-link" data-issue-key="LU-12901"><del>LU-12901</del></a> o2iblnd: retry qp creation with reduced queue depth<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8a3ef5713cc4aed1ac7bd3ce177895caa597cc4c</p> <p>Landed for 2.14</p> <p>"Etienne AUJAMES &lt;eaujames@ddn.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45901" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45901</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12901" title="Failing to create a properly sized IB queue pair" class="issue-link" data-issue-key="LU-12901"><del>LU-12901</del></a> o2iblnd: retry qp creation with reduced queue depth<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 9e0736f2306286f2f2c653c4e06c17d2201d1c0f</p> Related LU-10213 LU-7124 Development Rank 1|i00oin: Rank (Obsolete) 9223372036854775807 Severity