https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10020 Lustre <p>We have the patch from <a href="https://jira.whamcloud.com/browse/LU-8752" title="mlx5_warn:mlx5_0:dump_cqe:257:" class="issue-link" data-issue-key="LU-8752"><del>LU-8752</del></a> applied.</p> <p>We are testing lustre2.10.1 pre-release on a mlx5 hca host. lnet_selftest fails and mounting filesystem produced this error.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[ 435.503071] mlx5_warn:mlx5_1:dump_cqe:257:(pid 4031): dump error cqe [ 435.503072] 00000000 00000000 00000000 00000000 [ 435.503072] 00000000 00000000 00000000 00000000 [ 435.503073] 00000000 00000000 00000000 00000000 [ 435.503075] 00000000 9d005304 08000069 005878d2 [ 435.503078] LNet: 4031:0:(o2iblnd_cb.c:3475:kiblnd_complete()) RDMA (tx: ffffc90063356f28) failed: 4 [ 435.503292] LNet: 4029:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -&gt; 10.151.20.103@o2ib cookie 0x67 sending 1 waiting 0: failed 5 [ 435.503295] LNet: 4029:0:(o2iblnd_cb.c:1919:kiblnd_close_conn_locked()) Closing conn to 10.151.20.103@o2ib: error -5(waiting) [ 435.503304] LNet: 4029:0:(rpc.c:1413:srpc_lnet_ev_handler()) LNet event status -5 type 1, RPC errors 11 [ 435.503306] LNet: 4029:0:(rpc.c:1413:srpc_lnet_ev_handler()) Skipped 1 previous similar message [ 435.503396] LNet: 4151:0:(rpc.c:1143:srpc_client_rpc_done()) Client RPC done: service 5, peer 12345-10.151.20.103@o2ib, status SWI_STATE_REQUEST_SUBMITTED:1:-4 [ 440.503751] LNet: 4152:0:(lib-move.c:830:lnet_post_send_locked()) Dropping message <span class="code-keyword">for</span> 12345-10.151.20.103@o2ib: peer not alive [ 440.503754] LNet: 4152:0:(lib-move.c:2827:LNetPut()) Error sending PUT to 12345-10.151.20.103@o2ib: -113 [ 440.503757] LNet: 4152:0:(rpc.c:1413:srpc_lnet_ev_handler()) LNet event status -113 type 5, RPC errors 16 [ 440.503758] LNet: 4152:0:(rpc.c:1413:srpc_lnet_ev_handler()) Skipped 4 previous similar messages [ 440.503765] LNet: 4152:0:(rpc.c:1143:srpc_client_rpc_done()) Client RPC done: service 5, peer 12345-10.151.20.103@o2ib, status SWI_STATE_REQUEST_SUBMITTED:1:-4 [ 506.581347] LNet: 4173:0:(rpc.c:1069:srpc_client_rpc_expired()) Client RPC expired: service 11, peer 12345-10.151.20.103@o2ib, timeout 64. [ 506.581363] LNet: 4147:0:(rpc.c:1143:srpc_client_rpc_done()) Client RPC done: service 11, peer 12345-10.151.20.103@o2ib, status SWI_STATE_REQUEST_SENT:1:-4 [ 506.581367] LustreError: 4147:0:(brw_test.c:344:brw_client_done_rpc()) BRW RPC to 12345-10.151.20.103@o2ib failed with -110 </pre> </div></div> <p> </p> <p> </p> <p> </p> LU-10020

mlx5_warn:mlx5_1:dump_cqe:257:(pid 4031): dump error cqe

Bug Critical Resolved Not a Bug Amir Shehata Mahmoud Hanafi Fri, 22 Sep 2017 00:55:36 +0000 Tue, 1 Sep 2020 14:13:25 +0000 Tue, 18 Dec 2018 17:43:25 +0000 Lustre 2.10.0 0 7 <p>Hello,</p> <p>Can you please confirm you are using 2.10.1 pre-release on a production system?</p> <p>This particular system is a router and not in production yet. We are testing multirail for production use.</p> <p> </p> <p> </p> <p>Then let's move to sev 2 - sev 1 is just for production outages</p> <p>Amir</p> <p>Can you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Additional Info:</p> <p>HCA:  Mellanox Technologies MT27700 Family <span class="error">&#91;ConnectX-4&#93;</span></p> <p>OS: Sles12 SP2 4.4.74-92.32.1.20170808-nasa</p> <p>OFED: mlnx ofed 3.4.2</p> <p> </p> <p>Lustre 2.9 Works</p> <p>Lustre 2.10.0 Works.</p> <p>Lustre 2.10.1 Does not work</p> <p> </p> <p>I wonder if commit:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">commit f87c7c2cee6fc5a0864a757917a414dc605554b3 Author: Doug Oucharek &lt;doug.s.oucharek@intel.com&gt; Date: Tue May 16 16:00:53 2017 -0700 LU-9500 lnd: Don't Page Align remote_addr with FastReg </pre> </div></div> <p>Is the problem. </p> <p>Can you take out this commit from your 2.10.1 tree and try it out?</p> <p>I'll try and reproduce locally as well</p> <p>removing commit f87c7c2cee6fc5a0864a757917a414dc605554b3 fixed the problem in mofed3.4.2. We are building mofed4.1 and will test soon.<br/> It may be required with mofed4.x.</p> <p>Ok. <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> was intended to get mofed 4.1 working, but it shouldn't have broken mofed3.4.2. We'll need to resolve that.</p> <p>I tested mofed4.1 and lustre2.10.1. Didn't get the error.</p> <p>Just to clarify, is the passing test with mofed 4.1, Lustre 2.10.1 minus <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a>? Or with <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a>?</p> <p>I couldn't reproduce the failure with lnet-selftest on RHEL 7.3/MOFED 3.4.2/Lustre 2.10.1-RC1.<br/> are you able to consistently reproduce this with mofed3.4.2 + 2.10.1? This patch specifically addresses fastreg.</p> <p>would you be able to provide us with net/neterror logs for this problem?</p> <p>With Rhel7.3/Mofed3.4.2/lustre2.10.1-Rc1 I didn't always get the "dump error cqe." But lnet-selftest wasn't working with lots lnet errors.</p> <p>With Rhel7.3/Mofed3.4.2/lustre 2.10 and Rhel7.3/Mofed4.1/lustre 2.10.1-RC1 it always works</p> <p>And Removing <a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> with Rhel7.3/Mofed3.4.2/lustre2.10.1-Rc1 it always worked.</p> <p>Currently I am running with Rhel7.3/Mofed4.1/lustre 2.10.1-RC1. I'll will revert and gather some logs.</p> <p>I think to reproduce it map_on_demand needs to be configured<br/> Here is our config.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">options ko2iblnd timeout=150 retry_count=7 peer_timeout=0 map_on_demand=32 peer_credits=63 concurrent_sends=63 </pre> </div></div> <p>ok, I'm able to reproduce with map_on_demand=32.</p> <p>I'll investigate further.</p> <p>It looks like the issue is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">kiblnd_fmr_map_tx() ... <span class="code-keyword">if</span> (!is_fastreg) rd-&gt;rd_frags[0].rf_addr &amp;= ~hdev-&gt;ibh_page_mask; </pre> </div></div> <p>for MOFED 3.4.2 it appears that we need to page align the remote_addr even when fast_reg is enabled.</p> <p>Next step is to install MOFED 4.1 and see if page aligning the remote_address will trigger failure. If it does then we might need to check for MOFED version to see if we need to page align.</p> <p>Please look at <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a>. I believe this would be the same problem. <a href="https://review.whamcloud.com/29290" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29290</a> should resolve that issue.</p> <p>Let me know if the patch helps in your case.</p> <p>Will <a href="https://jira.whamcloud.com/browse/LU-9983" title="LBUG llog_osd.c:327:llog_osd_declare_write_rec() - all DNE MDS" class="issue-link" data-issue-key="LU-9983"><del>LU-9983</del></a> land as the solution for this issue? We have moved up to mofed4 so we are no longer seeing the issue. </p> <p> </p> <p>Please take a look at my comment here:<br/> <a href="https://jira.hpdd.intel.com/browse/LU-10089?focusedCommentId=210745&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-210745" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-10089?focusedCommentId=210745&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-210745</a></p> <p>think this can be close. All fixes has been pushed to mofed 4.4.2</p> <p>ok - thanks</p> Related Development Rank 1|hzzkm7: Rank (Obsolete) 9223372036854775807 Severity