https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14733 Lustre <p>lnet_selftest fails between two nodes over Omnipath</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>dk.opal63.llnl.gov.7:00000001:00020000:43.0:1622598261.714620:0:129525:0:(brw_test.c:415:brw_bulk_ready()) BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103 </pre> </div></div> <p>Bulk transfers work over Infiniband (although in that test 1 of the nodes was RHEL 7.9 and an earlier Lustre patch stack).  Bulk transfers also work over tcp using ksocklnd.</p> <p>lctl pings work fine between the same two nodes.</p> <p>mpibench and other MPI applications also work fine over Omnipath between two nodes.</p> <p>See <a href="https://github.com/LLNL/lustre/releases/tag/2.12.6_9.llnl" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/releases/tag/2.12.6_9.llnl</a> for the patch stack</p> lustre-2.12.6_9.llnl client <br/> kernel-4.18.0-305.0.0.1toss.t4.x86_64 <br/> RHEL84 LU-14733

brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103

Bug Critical Resolved Fixed Serguei Smirnov Olaf Faaland LTS12 llnl Thu, 3 Jun 2021 15:12:03 +0000 Fri, 18 Mar 2022 21:15:27 +0000 Sat, 24 Jul 2021 00:23:53 +0000 Lustre 2.12.8 Lustre 2.15.0 0 11 <p>These nodes were RHEL 8.3 based and have been upgraded to RHEL 8.4.  At the same time, they got an updated Lustre client with the following OS-compat patches:</p> <ul> <li>12d13f33c0 (tag: 2.12.6_9.llnl) <a href="https://jira.whamcloud.com/browse/LU-13783" title="Support for linux kernel version 5.8" class="issue-link" data-issue-key="LU-13783"><del>LU-13783</del></a> osc: handle removal of NR_UNSTABLE_NFS</li> <li>3cdb219927 <a href="https://jira.whamcloud.com/browse/LU-12355" title="Support for linux kernel version 5.0" class="issue-link" data-issue-key="LU-12355"><del>LU-12355</del></a> llite: MS_* flags and SB_* flags split</li> <li>03e48854cb <a href="https://jira.whamcloud.com/browse/LU-12355" title="Support for linux kernel version 5.0" class="issue-link" data-issue-key="LU-12355"><del>LU-12355</del></a> llite: totalram_pages changed to atomic_long_t</li> <li>b055fe9f7a (tag: 2.12.6_8.llnl) <a href="https://jira.whamcloud.com/browse/LU-14690" title="RHEL8.4 support" class="issue-link" data-issue-key="LU-14690"><del>LU-14690</del></a> kernel: new kernel <span class="error">&#91;RHEL 8.4 4.18.0-305.el8&#93;</span></li> <li>335e03049d (tag: 2.12.6_7.llnl) <a href="https://jira.whamcloud.com/browse/LU-14673" title="panic: crc32-table: crc32 alg self test failed in fips mode!" class="issue-link" data-issue-key="LU-14673"><del>LU-14673</del></a> sec: annotate algorithms taking optional key</li> </ul> <p>See also <a href="https://jira.whamcloud.com/browse/LU-14690" title="RHEL8.4 support" class="issue-link" data-issue-key="LU-14690"><del>LU-14690</del></a></p> <p>Serguei</p> <p>Can you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>For my reference, my local ticket is TOSS-5228</p> <p>Hi Serguei, do you have any update or questions on this? Thanks</p> <p>Olaf,</p> <p>So far it looks like it is possible there's some sort of incompatibility in how ib_post_send is called, but I don't have anything concrete in this direction yet. Do you have any config logs (from building lustre)? I'm not that familiar with Omnipath either. Which version of Omnipath are you using? Basically I want to make sure we end up calling ib_post_send correctly.</p> <p>Thanks,</p> <p>Serguei.</p> <p>Hi Serguei,<br/> I've attached <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/39055/39055_build.txt" title="build.txt attached to LU-14733">build.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> - not the config.log, but at least the stdout from ./configure. I'll look into Omnipath info.<br/> thanks,<br/> Olaf</p> <p>Hi Olaf,</p> <p>Here's some more detail regarding what I'd like to try with the OPA build:</p> <p>lnet/autoconf/lustre-lnet.m4 has a check for ib_post_send() and ib_post_recv() to see if they require const ptr parameters. Could you please try removing this check and build without it? (See attached diff file). I suspect there may be an issue with using the wrong header file when building. The kernel code for 4.18.0 appears to define these functions without the "const" and I think that's what we should be using for OPA, but the stdout you provided indicates that as a result of the configure the "const" version is used.<span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/39075/39075_diff.txt" title="diff.txt attached to LU-14733">diff.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>Thanks,</p> <p>Serguei.</p> <p>Hi Serguei,</p> <p>Sorry, I didn't see your message for some reason. My test was slightly different, but I think still produced the result you need. With the config check sabotaged so HAVE_IB_POST_SEND_RECV_CONST is not defined, the build fails with</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>/g/g0/faaland1/rpmbuild/BUILD/lustre-2.12.6_9.llnl_2_gd3006c6/lnet/klnds/o2iblnd/o2iblnd_cb.c:1002:46: error: passing argument 3 of 'ib_post_send' from incompatible pointer type [-Werror=incompatible-pointer-types] rc = ib_post_send(conn-&gt;ibc_cmid-&gt;qp, wr, &amp;bad); ^~~~ In file included from /usr/src/kernels/4.18.0-305.0.0.1toss.t4.x86_64/include/rdma/ib_addr.h:20, from /usr/src/kernels/4.18.0-305.0.0.1toss.t4.x86_64/include/rdma/rdma_cm.h:12, from /g/g0/faaland1/rpmbuild/BUILD/lustre-2.12.6_9.llnl_2_gd3006c6/lnet/klnds/o2iblnd/o2iblnd.h:71, from /g/g0/faaland1/rpmbuild/BUILD/lustre-2.12.6_9.llnl_2_gd3006c6/lnet/klnds/o2iblnd/o2iblnd_cb.c:37: /usr/src/kernels/4.18.0-305.0.0.1toss.t4.x86_64/include/rdma/ib_verbs.h:3799:37: note: expected 'const struct ib_send_wr **' but argument is of type 'struct ib_send_wr **' const struct ib_send_wr **bad_send_wr) ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~ </pre> </div></div> <p>My change was</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/lnet/autoconf/lustre-lnet.m4 b/lnet/autoconf/lustre-lnet.m4 index 6d4a05d4ef..98490ea9ae 100644 --- a/lnet/autoconf/lustre-lnet.m4 +++ b/lnet/autoconf/lustre-lnet.m4 @@ -539,6 +539,8 @@ AS_IF([test $ENABLEO2IB != "no"], [ # In MOFED 4.6, the second and third parameters for # ib_post_send() and ib_post_recv() are declared with # 'const'. + # + # SABOTAGE: force this to fail with extra argument to ib_post_send tmp_flags="$EXTRA_KCFLAGS" EXTRA_KCFLAGS="-Werror" LB_CHECK_COMPILE([if 'ib_post_send() and ib_post_recv()' have const parameters], @@ -555,7 +557,7 @@ AS_IF([test $ENABLEO2IB != "no"], [ #include &lt;rdma/ib_verbs.h&gt; ],[ ib_post_send(NULL, (const struct ib_send_wr *)NULL, - (const struct ib_send_wr **)NULL); + (const struct ib_send_wr **)NULL, NULL); ],[ AC_DEFINE(HAVE_IB_POST_SEND_RECV_CONST, 1, [ib_post_send and ib_post_recv have const parameters]) </pre> </div></div> <p>thanks,<br/> Olaf</p> <p>Olaf,</p> <p>I suppose this means we need to look for a different reason why ib_post_send would fail with EINVAL (22) error code.</p> <p>So far I wasn't able to find the source code for this function provided by Omnipath for RH8.4. Oddly, I also failed to locate Omnipath release notes for RH8.4. Perhaps someone from CN can comment.</p> <p>Btw, although this is unlikely to be related to this issue, perhaps you want to port <del><a href="https://jira.whamcloud.com/browse/LU-13182" title="MAP_POPULATE hangs with Linux 5.4" class="issue-link" data-issue-key="LU-13182"><del>LU-13182</del></a></del> as well? (see last couple of comments in  <a href="https://jira.whamcloud.com/browse/LU-14690" title="RHEL8.4 support" class="issue-link" data-issue-key="LU-14690"><del>LU-14690</del></a>)</p> <p>Thanks,</p> <p>Serguei.</p> <p>Hi Serguei, I'm told:</p> <p>"OPA hardware does not have hardware verbs support, so it uses the rdmavt layer to provide software translation. You want to look at rvt_post_send() in drivers/infiniband/sw/rdmavt/qp.c."</p> <p>We're using the in-kernel OPA driver, so in this case the one in linux-4.18.0-305.el8</p> <p>thanks</p> <p>Olaf,</p> <p>Have you tried building lustre master and testing if it has the same problem on your RH8.4 OPA machine? There's a chance that one of the o2iblnd patches there may help. Otherwise I'm thinking I'm going to have to add some debug messages to LLNL patch stack so we can use that to get more data.</p> <p>Thanks,</p> <p>Serguei.</p> <p>Just to document this where it's visible to everyone:</p> <p>Ran perftest (perftest-4.4-37.0.t4.x86_64) from opal63 (compute), used both opal64 (compute) and opal187 (router) as servers. These are the same nodes, running the same OS, where lnet_selftest fails.</p> <p>All these tests succeeded:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ib_write_bw opal64-hsi0 ib_read_bw opal64-hsi0 ib_send_bw opal64-hsi0 ib_atomic_bw opal64-hsi0 ib_write_lat opal187-hsi0 ib_send_bw -a -b -R -F opal187-hsi0 ib_send_bw -a -b -R -F opal64-hsi0 ib_send_bw -a -b -R -F -q 3 opal64-hsi0 ib_send_bw -a -b -R -F -q 3 opal187-hsi0 ib_read_bw -a -b -R -F -q 3 opal187-hsi0 ib_atomic_bw -b -R -F -q 3 opal187-hsi0 ib_send_bw -a -b -R -F -q 3 opal187-hsi0 ib_write_bw -a -b -R -F -q 3 opal187-hsi0 ib_read_bw -a -b -R -F -q 3 opal187-hsi0 </pre> </div></div> <p>Those arguments mean:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> -R, --rdma_cm Connect QPs with rdma_cm and run test on those QPs -z, --com_rdma_cm Communicate with rdma_cm module to exchange data - use regular QPs -m, --mtu=&lt;mtu&gt; QP Mtu size (default: active_mtu from ibv_devinfo) -c, --connection=&lt;type&gt; Connection type RC/UC/UD/XRC/DC/SRD (default RC). -d, --ib-dev=&lt;dev&gt; Use IB device &lt;dev&gt; (default: first device found) -i, --ib-port=&lt;port&gt; Use network port &lt;port&gt; of IB device (default: 1) -s, --size=&lt;size&gt; Size of message to exchange (default: 1) -a, --all Run sizes from 2 till 2^23 -n, --iters=&lt;iters&gt; Number of exchanges (at least 100, default: 1000) -x, --gid-index=&lt;index&gt; Test uses GID with GID index taken from command -V, --version Display version number -e, --events Sleep on CQ events (default poll) -F, --CPU-freq Do not fail even if cpufreq_ondemand module -I, --inline_size=&lt;size&gt; Max size of message to be sent in inline mode -u, --qp-timeout=&lt;timeout&gt; QP timeout = (4 uSec)*(2^timeout) (default: 14) -S, --sl=&lt;sl&gt; Service Level (default 0) </pre> </div></div> <p>Hi Serguei,</p> <blockquote><p>Have you tried building lustre master and testing if it has the same problem on your RH8.4 OPA machine?</p></blockquote> <p>We were in the process of setting that up when we lost power to the machine.  As soon as it's back up, we will test that.</p> <p>We've now tried lnet_selftest on the OPA compute nodes opal63 and opal64 using lustre-2.14.0_2.llnl <a href="https://github.com/LLNL/lustre/tree/2.14.0_2.llnl." class="external-link" target="_blank" rel="nofollow noopener">https://github.com/LLNL/lustre/tree/2.14.0_2.llnl.</a> The results were the same as with lustre-2.12.6_9.llnl, that is, the test failed and we saw bad numbers for the results (lots of 0s) and the same error in the kernel dumps as before:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (o2iblnd_cb.c:957:kiblnd_post_tx_locked()) Error -22 posting transmit to 192.168.128.5@o2ib18 </pre> </div></div> <p>We're in the process of repeating the same test using lustre master.</p> <p>The test with master hit a list corruption BUG, so it was inconclusive.  I won't be able to track it down now, I'm leaving for vacation soon and need to prepare.</p> <p>Can someone post instructions or a test script so we can take a look? I downloaded the LLNL lustre tarball and built it on a RHEL 8.4 box (stock kernel) and would like to give it a shot, but not very familiar with Lnet self test.</p> <p>Dennis,</p> <p>There's instructions here: <a href="https://wiki.lustre.org/LNET_Selftest" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.lustre.org/LNET_Selftest</a></p> <p>Basically, two nodes with LNet running are needed</p> <p>1) Make sure selftest is loaded on both nodes: modprobe lnet_selftest</p> <p>2) Make sure LNet is configured on both nodes. Run "lnetctl net show" to list local nids on each.</p> <p>3) Make the nodes "discover" each other: "lnetctl discover &lt;peer nid&gt;"</p> <p>4) Copy the wrapper script on one of the nodes. Use primary nids to fill in "TO" and "FROM"</p> <p>5) Run the script</p> <p> </p> <p>The 305 kernel may be the first RHEL kernel with fmr removed...</p> <p>Am able to reproduce an EINVAL error internally. Looks like some of the post_send calls work but the ones from kiblnd_sd always fail. Here is a kprobe that I used to dump the return value of rvt_post_send():</p> <p>r:testprobe rdmavt:rvt_post_send $retval</p> <p>These threads always succeed:</p> <p>lst_t_00_24-7492 <span class="error">&#91;039&#93;</span> d... 88520.827471: testprobe: (kiblnd_post_tx_locked+0x857/0xa50 <span class="error">&#91;ko2iblnd&#93;</span> &lt;- rvt_post_send) arg1=0x0</p> <p>kiblnd_connd-7547 <span class="error">&#91;000&#93;</span> d... 88520.827888: testprobe: (ib_send_mad+0x235/0x420 <span class="error">&#91;ib_core&#93;</span> &lt;- rvt_post_send) arg1=0x0</p> <p>monitor_thread-7556 <span class="error">&#91;032&#93;</span> d... 88521.503931: testprobe: (kiblnd_post_tx_locked+0x857/0xa50 <span class="error">&#91;ko2iblnd&#93;</span> &lt;- rvt_post_send) arg1=0x0</p> <p>These threads always fail:</p> <p>kiblnd_sd_00_01-7549 <span class="error">&#91;014&#93;</span> d... 88520.827678: testprobe: (kiblnd_post_tx_locked+0x857/0xa50 <span class="error">&#91;ko2iblnd&#93;</span> &lt;- rvt_post_send) arg1=0xffffffea</p> <p>Note 0xffffffea decodes to -22 in decimal.</p> <p> </p> <p>The -EINVAL is happening because a post_send of a IB_WR_LOCAL_INV operation is failing.   So this is an issue with the fastreg stuff as I expected.</p> <p><span class="error">&#91;^kprobes-off.sh,&#93;</span> <span class="error">&#91;^kprobes.sh,&#93;</span> and <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/39289/39289_trace1.txt" title="trace1.txt attached to LU-14733">trace1.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> contain the kprobes used and the tracing.</p> <p>I'm trying to get more details now.</p> <p> </p> <p>I should point our our upstream CI testing validates NFS/RDMA, iSer, SRP use of the fast reg feature and has been passing without issue.</p> <p>I have attached refined kprobes and trace2.txt.</p> <p>Here is the invalidate code:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-object">int</span> rvt_invalidate_rkey(struct rvt_qp *qp, u32 rkey) { &lt;snip&gt; mr = rcu_dereference( rkt-&gt;table[(rkey &gt;&gt; (32 - dev-&gt;dparms.lkey_table_size))]); <span class="code-keyword">if</span> (unlikely(!mr || mr-&gt;lkey != rkey || qp-&gt;ibqp.pd != mr-&gt;pd)) *** rkey is != mr-&gt;lkey <span class="code-keyword">goto</span> bail; &lt;snip&gt; bail: rcu_read_unlock(); <span class="code-keyword">return</span> -EINVAL; } </pre> </div></div> <p>Here is an excerpt from the trace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>*** focusing on keys that being with 0x16f700 lst_t_00_21-7489 [011] d... 426264.174874: rvt_invalidate_rkey_p: (rvt_invalidate_rkey+0x0/0x60 [rdmavt]) qpn=0xe0 pd=0xffffa0ab4831b780 rkey=0x16f7000 lst_t_00_21-7489 [011] d.Z. 426264.174879: rvt_invalidate_rkey_1: (rvt_invalidate_rkey+0x26/0x60 [rdmavt]) mr=0xffffc0dfca0d3b78 table_ptr=0xffffc0dfca0d3b78 lst_t_00_21-7489 [011] d... 426264.174881: rvt_invalidate_rkey_2: (rvt_invalidate_rkey+0x29/0x60 [rdmavt]) rkey=0x16f7000 mr_lkey=0x16f7000 lst_t_00_21-7489 [011] d... 426264.174882: rvt_invalidate_rkey: (rvt_post_send+0x525/0x800 [rdmavt] &lt;- rvt_invalidate_rkey) ret=0x0 *** 0x16f7000 has been invalidated *** lst_t_00_21-7489 [011] d... 426264.174884: rvt_fast_reg_mr_p: (rvt_fast_reg_mr+0x0/0x70 [rdmavt]) qpn=0xe0 ibmr=0xffffa0a316a16a00 pd=0xffffa0ab4831b780 key=0x16f7001 lst_t_00_21-7489 [011] d... 426264.174886: rvt_fast_reg_mr: (rvt_post_send+0x1a3/0x800 [rdmavt] &lt;- rvt_fast_reg_mr) ret=0x0 *** 0x16f7001 has been written into ibmr.mr keys because of the above fast reg *** *** then an invalidate is posted for 0x16f7000 kiblnd_sd_00_03-7551 [015] d... 426264.175096: rvt_invalidate_rkey_p: (rvt_invalidate_rkey+0x0/0x60 [rdmavt]) qpn=0xe0 pd=0xffffa0ab4831b780 rkey=0x16f7000 kiblnd_sd_00_03-7551 [015] d.Z. 426264.175100: rvt_invalidate_rkey_1: (rvt_invalidate_rkey+0x26/0x60 [rdmavt]) mr=0xffffc0dfca0d3b78 table_ptr=0xffffc0dfca0d3b78 *** the key in the mr is the one fastreg'ed from 426264.174884 kiblnd_sd_00_03-7551 [015] d... 426264.175101: rvt_invalidate_rkey_2: (rvt_invalidate_rkey+0x29/0x60 [rdmavt]) rkey=0x16f7000 mr_lkey=0x16f7001 kiblnd_sd_00_03-7551 [015] d... 426264.175103: rvt_invalidate_rkey: (rvt_post_send+0x525/0x800 [rdmavt] &lt;- rvt_invalidate_rkey) ret=0xffffffea kiblnd_sd_00_03-7551 [015] d.Z. 426264.175104: rvt_post_send_err1: (rvt_post_send+0x475/0x800 [rdmavt]) wr=0xffffa0a2e7da7ed0 wr_opcode=7 err=-22 *** and the post send fails *** kiblnd_sd_00_03-7551 [015] d... 426264.175105: rvt_post_send: (kiblnd_post_tx_locked+0x857/0xa50 [ko2iblnd] &lt;- rvt_post_send) ret=0xffffffea </pre> </div></div> <p>It looks to me like Lustre is losing track of the keys for a particular MR?</p> <p>Thanks, Mike and Dennis.</p> <p>Serguei, please let us know when you (or someone) are working on this new information Cornelis came up with.  Thank you.</p> <p>It looks like Lustre is sending a gratuitous invalidate because of this code fragment:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* There appears to be a bug in MLX5 code where you must * invalidate the rkey of a <span class="code-keyword">new</span> FastReg pool before first * using it. Thus, I am marking the FRD invalid here. */ frd-&gt;frd_valid = <span class="code-keyword">false</span>; </pre> </div></div> <p>This is not wrong, but difference than other ULPs.</p> <p>The following code is then executed before any fast reg has happened:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (!frd-&gt;frd_valid) { struct ib_rdma_wr *inv_wr; __u32 key = is_rx ? mr-&gt;rkey : mr-&gt;lkey; inv_wr = &amp;frd-&gt;frd_inv_wr; memset(inv_wr, 0, sizeof(*inv_wr)); inv_wr-&gt;wr.opcode = IB_WR_LOCAL_INV; inv_wr-&gt;wr.wr_id = IBLND_WID_MR; inv_wr-&gt;wr.ex.invalidate_rkey = key; <span class="code-comment">/* Bump the key */</span> key = ib_inc_rkey(key); *** updates keys in ib_mr, but not the rvt_mregion keys *** ib_update_fast_reg_key(mr, key); } </pre> </div></div> <p>The following code uses <b>struct rvt_mregion keys</b> to validate and doesn't see the above key change in the ibmr and fails the invalidate.   The rkey is correct, but the mr-&gt;lkey hasn't changed to match until the next</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-object">int</span> rvt_invalidate_rkey(struct rvt_qp *qp, u32 rkey) { struct rvt_dev_info *dev = ib_to_rvt(qp-&gt;ibqp.device); struct rvt_lkey_table *rkt = &amp;dev-&gt;lkey_table; struct rvt_mregion *mr; <span class="code-keyword">if</span> (rkey == 0) <span class="code-keyword">return</span> -EINVAL; rcu_read_lock(); mr = rcu_dereference( rkt-&gt;table[(rkey &gt;&gt; (32 - dev-&gt;dparms.lkey_table_size))]); <span class="code-keyword">if</span> (unlikely(!mr || mr-&gt;lkey != rkey || qp-&gt;ibqp.pd != mr-&gt;pd)) <span class="code-keyword">goto</span> bail; atomic_set(&amp;mr-&gt;lkey_invalid, 1); rcu_read_unlock(); <span class="code-keyword">return</span> 0; bail: rcu_read_unlock(); <span class="code-keyword">return</span> -EINVAL; } </pre> </div></div> <p>I'm working on the following fix:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>diff --git a/drivers/infiniband/sw/rdmavt/mr.c b/drivers/infiniband/sw/rdmavt/mr.c index 601d18dd..528727f 100644 --- a/drivers/infiniband/sw/rdmavt/mr.c +++ b/drivers/infiniband/sw/rdmavt/mr.c @@ -691,6 +691,7 @@ int rvt_invalidate_rkey(struct rvt_qp *qp, u32 rkey) struct rvt_dev_info *dev = ib_to_rvt(qp-&gt;ibqp.device); struct rvt_lkey_table *rkt = &amp;dev-&gt;lkey_table; struct rvt_mregion *mr; + struct rvt_mr *rmr; if (rkey == 0) return -EINVAL; @@ -698,7 +699,11 @@ int rvt_invalidate_rkey(struct rvt_qp *qp, u32 rkey) rcu_read_lock(); mr = rcu_dereference( rkt-&gt;table[(rkey &gt;&gt; (32 - dev-&gt;dparms.lkey_table_size))]); - if (unlikely(!mr || mr-&gt;lkey != rkey || qp-&gt;ibqp.pd != mr-&gt;pd)) + if (unlikely(!mr || qp-&gt;ibqp.pd != mr-&gt;pd)) + goto bail; + /* isolate parent */ + rmr = container_of(mr, struct rvt_mr, mr); + if (rmr-&gt;ibmr.type != IB_MR_TYPE_MEM_REG || rmr-&gt;ibmr.rkey != rkey) goto bail; atomic_set(&amp;mr-&gt;lkey_invalid, 1); </pre> </div></div> <p>Hi,</p> <p>To me this looks like a very good candidate to fix the issue. Thanks for taking the time to look into this!</p> <p>Thanks,</p> <p>Serguei.</p> <p>&gt; To me this looks like a very good candidate to fix the issue.</p> <p>Another potential fix is to delete the MLX5 hack.   That should work as well.</p> <p>The patch didn't work.</p> <p>I need to do more analysis.</p> <p>Hi Serguei,</p> <p>Do you (or Amir, or anyone else you have easy access to) know if the MLX issue that prompted that hack has been fixed? The JIRA issue was <a href="https://jira.whamcloud.com/browse/LU-8752" title="mlx5_warn:mlx5_0:dump_cqe:257:" class="issue-link" data-issue-key="LU-8752"><del>LU-8752</del></a> and the commit was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 783428b60a98874b4783f8da48c66019d68d84d6 Author: Doug Oucharek &lt;doug.s.oucharek@intel.com&gt; Date:   Mon Dec 12 09:31:37 2016 -0800     LU-8752 lnet: Stop MLX5 triggering a dump_cqe          We have found that MLX5 will trigger a dump_cqe if we don't     invalidate the rkey on a newly alloated MR for FastReg usage.          This fix just tags the MR as invalid on its creation if we are     using FastReg and that will force it to do an invalidate of the     rkey on first usage.      ... diff --git a/lnet/klnds/o2iblnd/o2iblnd.c b/lnet/klnds/o2iblnd/o2iblnd.c index e919008d44..ee5a01f9fa 100644 --- a/lnet/klnds/o2iblnd/o2iblnd.c +++ b/lnet/klnds/o2iblnd/o2iblnd.c @@ -1536,7 +1536,10 @@ static int kiblnd_alloc_freg_pool(kib_fmr_poolset_t *fps, kib_fmr_pool_t *fpo)                         goto out_middle;                 } -               frd-&gt;frd_valid = true; +               /* There appears to be a bug in MLX5 code where you must +                * invalidate the rkey of a new FastReg pool before first +                * using it. Thus, I am marking the FRD invalid here. */ +               frd-&gt;frd_valid = false;                 list_add_tail(&amp;frd-&gt;frd_list, &amp;fpo-&gt;fast_reg.fpo_pool_list);                 fpo-&gt;fast_reg.fpo_pool_size++; </pre> </div></div> <p>I reverted</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LU-8752 lnet: Stop MLX5 triggering a dump_cqe</pre> </div></div> <p>and tested. Initially I had nonzero bandwidth, which is different than I recall seeing before. After a few seconds the bandwidth recorded went to 0. lctl dk shows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000800:00020000:16.0F:1625017325.639837:0:514295:0:(o2iblnd_cb.c:1031:kiblnd_post_tx_locked()) Error -22 posting transmit to 192.168.128.3@o2ib18 00000800:00020000:6.0F:1625017325.639911:0:514296:0:(o2iblnd_cb.c:1031:kiblnd_post_tx_locked()) Error -22 posting transmit to 192.168.128.3@o2ib18 00000800:00000100:6.0:1625017325.639916:0:514296:0:(o2iblnd_cb.c:2101:kiblnd_close_conn_locked()) Closing conn to 192.168.128.3@o2ib18: error -22(waiting) 00000400:00000100:8.0F:1625017325.639943:0:514293:0:(rpc.c:1418:srpc_lnet_ev_handler()) LNet event status -22 type 5, RPC errors 1 00000400:00000100:6.0:1625017325.639947:0:514296:0:(rpc.c:1418:srpc_lnet_ev_handler()) LNet event status -22 type 5, RPC errors 2 00000001:00020000:7.0F:1625017325.639951:0:514856:0:(brw_test.c:415:brw_bulk_ready()) BRW bulk READ failed for RPC from 12345-192.168.128.3@o2ib18: -22 00000001:00020000:53.0:1625017325.639966:0:514858:0:(brw_test.c:415:brw_bulk_ready()) BRW bulk READ failed for RPC from 12345-192.168.128.3@o2ib18: -22 00000400:00000100:53.0:1625017325.639968:0:514858:0:(rpc.c:905:srpc_server_rpc_done()) Server RPC 000000007f1b43fb done: service brw_test, peer 12345-192.168.128.3@o2ib18, status SWI_STATE_BULK_STARTED:-5 00000001:00020000:53.0:1625017325.639971:0:514858:0:(brw_test.c:389:brw_server_rpc_done()) Bulk transfer to 12345-192.168.128.3@o2ib18 has failed: -5 00000800:00000100:17.0:1625017325.640281:0:514294:0:(o2iblnd_cb.c:3734:kiblnd_complete()) RDMA (tx: 0000000005d7d115) failed: 5 00000800:00020000:6.0:1625017325.640283:0:514296:0:(o2iblnd_cb.c:1031:kiblnd_post_tx_locked()) Error -22 posting transmit to 192.168.128.3@o2ib18 00000800:00000100:6.0:1625017325.640284:0:514296:0:(o2iblnd_cb.c:2101:kiblnd_close_conn_locked()) Closing conn to 192.168.128.3@o2ib18: error -22(waiting) 00000400:00000100:8.0:1625017325.640285:0:514293:0:(lib-msg.c:698:lnet_attempt_msg_resend()) msg 0@&lt;0:0&gt;-&gt;192.168.128.3@o2ib18 exceeded retry count 0 00000800:00020000:17.0:1625017325.640286:0:514294:0:(o2iblnd_cb.c:1031:kiblnd_post_tx_locked()) Error -22 posting transmit to 192.168.128.3@o2ib18 00000400:00000100:8.0:1625017325.640287:0:514293:0:(rpc.c:1418:srpc_lnet_ev_handler()) LNet event status -5 type 5, RPC errors 3 00000800:00000100:6.0:1625017325.640288:0:514296:0:(o2iblnd_cb.c:3734:kiblnd_complete()) RDMA (tx: 0000000029f85d1a) failed: 5 00000400:00000100:17.0:1625017325.640290:0:514294:0:(lib-msg.c:698:lnet_attempt_msg_resend()) msg 0@&lt;0:0&gt;-&gt;192.168.128.3@o2ib18 exceeded retry count 0 </pre> </div></div> <p>I do know that patch proposed is just wrong.    Testing against the <b>struct rvt_mregion lkey</b> <em>should</em> be correct.</p> <p>I need to add another kprobe with the original kernel to look at the keys for all ib_alloc_mr() allocations from birth and track from that point to failures.</p> <p>A portion of the comment says:</p> <blockquote><p>This fix just tags the MR as invalid on its creation if we are<br/> using FastReg and that will force it to do an invalidate of the<br/> rkey on first usage.</p></blockquote> <p>The inference is that the invalidate only happens on first MR use, but I don't see anywhere that sets <b>frd_invalid</b> to true?   It looks like it will happen all the time for all MRs.</p> <p>It is starting to look to me like there is a concurency issue where somehow the old key is subsequently passed to an invalidate.</p> <p>Here is an invalidate for rkey 0x100:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lst_t_00_09-3288 [046] d... 6592.029697: rvt_invalidate_rkey_p: (rvt_invalidate_rkey+0x0/0x60 [rdmavt]) qpn=0x8 pd=0xffff9ebe1a48a200 rkey=0x100 lst_t_00_09-3288 [046] d... 6592.029700: rvt_invalidate_rkey_1: (rvt_invalidate_rkey+0x29/0x60 [rdmavt]) table_ptr=0xffffada64a4d1000 lst_t_00_09-3288 [046] d.Z. 6592.029702: rvt_invalidate_rkey_2: (rvt_invalidate_rkey+0x36/0x60 [rdmavt]) mr_lkey=0x100 ib_mr_type=0 ib_mr_rkey=0x101 ib_mr_lkey=0x101 lst_t_00_09-3288 [046] d... 6592.029705: rvt_invalidate_rkey: (rvt_post_send+0x525/0x800 [rdmavt] &lt;- rvt_invalidate_rkey) ret=0x0 </pre> </div></div> <p>Here is the one that fails:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kiblnd_sd_00_00-3359 [036] d... 6592.029947: rvt_invalidate_rkey_p: (rvt_invalidate_rkey+0x0/0x60 [rdmavt]) qpn=0x8 pd=0xffff9ebe1a48a200 rkey=0x100 lst_t_00_13-3292 [050] d... 6592.029947: rvt_lkey_ok_p: (rvt_lkey_ok+0x0/0x380 [rdmavt]) pd=0xffff9ebe1a48a200 sge_lkey=0x0 lst_t_00_18-3297 [001] d.Z. 6592.029948: rvt_invalidate_rkey_2: (rvt_invalidate_rkey+0x36/0x60 [rdmavt]) mr_lkey=0x60700 ib_mr_type=0 ib_mr_rkey=0x60701 ib_mr_lkey=0x60701 lst_t_00_13-3292 [050] d... 6592.029948: rvt_lkey_ok: (rvt_post_send+0x2dc/0x800 [rdmavt] &lt;- rvt_lkey_ok) ret=0x1 kiblnd_sd_00_00-3359 [036] d... 6592.029950: rvt_invalidate_rkey_1: (rvt_invalidate_rkey+0x29/0x60 [rdmavt]) table_ptr=0xffffada64a4d1000 lst_t_00_18-3297 [001] d... 6592.029951: rvt_invalidate_rkey: (rvt_post_send+0x525/0x800 [rdmavt] &lt;- rvt_invalidate_rkey) ret=0x0 kiblnd_sd_00_00-3359 [036] d.Z. 6592.029952: rvt_invalidate_rkey_2: (rvt_invalidate_rkey+0x36/0x60 [rdmavt]) mr_lkey=0x101 ib_mr_type=0 ib_mr_rkey=0x101 ib_mr_lkey=0x101 lst_t_00_13-3292 [050] dN.. 6592.029952: rvt_post_send: (kiblnd_post_tx_locked+0x857/0xa50 [ko2iblnd] &lt;- rvt_post_send) ret=0x0 lst_t_00_18-3297 [001] d... 6592.029952: rvt_fast_reg_mr_p: (rvt_fast_reg_mr+0x0/0x70 [rdmavt]) qpn=0x4 ibmr=0xffff9ebe3fd0e600 pd=0xffff9ebe1a48a200 key=0x60701 &lt;idle&gt;-0 [010] d.h. 6592.029952: rvt_lkey_ok_p: (rvt_lkey_ok+0x0/0x380 [rdmavt]) pd=0xffff9ebe1a48a200 sge_lkey=0x0 &lt;idle&gt;-0 [010] d.h. 6592.029953: rvt_lkey_ok: (rvt_get_rwqe+0x2c8/0x450 [rdmavt] &lt;- rvt_lkey_ok) ret=0x1 lst_t_00_12-3291 [049] dN.. 6592.029953: rvt_invalidate_rkey_p: (rvt_invalidate_rkey+0x0/0x60 [rdmavt]) qpn=0xa pd=0xffff9ebe1a48a200 rkey=0x70800 lst_t_00_18-3297 [001] d... 6592.029954: rvt_fast_reg_mr: (rvt_post_send+0x1a3/0x800 [rdmavt] &lt;- rvt_fast_reg_mr) ret=0x0 kiblnd_sd_00_00-3359 [036] d... 6592.029955: rvt_invalidate_rkey: (rvt_post_send+0x525/0x800 [rdmavt] &lt;- rvt_invalidate_rkey) ret=0xffffffea </pre> </div></div> <p>Note the different CPU and task name.</p> <p>Hi Mike and Serguei,</p> <p>Do you have any update on this issue?</p> <p>I'm expecting to get my OPA test cluster back again today, at which point I plan to more thoroughly compare the behavior of lnet_selftest both with and without the patch to invalidate the rkey before the mr is used.</p> <p>thanks,</p> <p>Olaf</p> <p>There are two issues with the existing code and hfi1.</p> <p>First there is what I suspect is a misplaced racy assignment:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> void kiblnd_fmr_pool_unmap(struct kib_fmr *fmr, <span class="code-object">int</span> status) { &lt;snip&gt; This code returns the frd to the list <span class="code-keyword">if</span> (frd) { frd-&gt;frd_valid = <span class="code-keyword">false</span>; spin_lock(&amp;fps-&gt;fps_lock); list_add_tail(&amp;frd-&gt;frd_list, &amp;fpo-&gt;fast_reg.fpo_pool_list); spin_unlock(&amp;fps-&gt;fps_lock); fmr-&gt;fmr_frd = NULL; &lt;- I think <span class="code-keyword">this</span> should be before add } &lt;snip&gt; I suspect the NULL should be before adding to the list to avoid a race with the map &lt;snip&gt; pulling the descriptor from the list } </pre> </div></div> <p>The mapping looks like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-object">int</span> kiblnd_fmr_pool_map(struct kib_fmr_poolset *fps, struct kib_tx *tx, struct kib_rdma_desc *rd, u32 nob, u64 iov, struct kib_fmr *fmr) { &lt;snip&gt;This code dequeues the kib_fast_reg_descriptor from a list struct kib_fast_reg_descriptor *frd; #ifdef HAVE_IB_MAP_MR_SG struct ib_reg_wr *wr; <span class="code-object">int</span> n; #<span class="code-keyword">else</span> struct ib_rdma_wr *wr; struct ib_fast_reg_page_list *frpl; #endif struct ib_mr *mr; frd = list_first_entry(&amp;fpo-&gt;fast_reg.fpo_pool_list, struct kib_fast_reg_descriptor, frd_list); list_del(&amp;frd-&gt;frd_list); spin_unlock(&amp;fps-&gt;fps_lock); &lt;snip&gt; This code sets up the invalidate operation embbeded in the kib_fast_reg_descriptor <span class="code-keyword">if</span> (!frd-&gt;frd_valid) { struct ib_rdma_wr *inv_wr; __u32 key = is_rx ? mr-&gt;rkey : mr-&gt;lkey; inv_wr = &amp;frd-&gt;frd_inv_wr; memset(inv_wr, 0, sizeof(*inv_wr)); inv_wr-&gt;wr.opcode = IB_WR_LOCAL_INV; inv_wr-&gt;wr.wr_id = IBLND_WID_MR; inv_wr-&gt;wr.ex.invalidate_rkey = key; <span class="code-comment">/* Bump the key */</span> key = ib_inc_rkey(key); ib_update_fast_reg_key(mr, key); } &lt;snip&gt; The code goes on to register the pages in ib_map_mr_sg and sets up the &lt;snip&gt; frd_fastreg_wr embedded in the kib_fast_reg_descriptor &lt;snip&gt; &lt;snip&gt; The code then fuses the kib_fast_reg_descriptor kib_fmr in the kib_tx fmr-&gt;fmr_key = is_rx ? mr-&gt;rkey : mr-&gt;lkey; fmr-&gt;fmr_frd = frd; &lt;--- here fmr-&gt;fmr_pool = fpo; <span class="code-keyword">return</span> 0; &lt;snip&gt; At <span class="code-keyword">this</span> point no posts have been done and they are defered to &lt;snip&gt; kiblnd_post_tx_locked(). } </pre> </div></div> <p>The post send looks like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> <span class="code-object">int</span> kiblnd_post_tx_locked(struct kib_conn *conn, struct kib_tx *tx, <span class="code-object">int</span> credit) __must_hold(&amp;conn-&gt;ibc_lock) { &lt;snip&gt; This code sees frd from above and prepends WRs from the kib_fast_reg_descriptor <span class="code-keyword">if</span> (frd != NULL) { <span class="code-keyword">if</span> (!frd-&gt;frd_valid) { wr = &amp;frd-&gt;frd_inv_wr.wr; wr-&gt;next = &amp;frd-&gt;frd_fastreg_wr.wr; } <span class="code-keyword">else</span> { wr = &amp;frd-&gt;frd_fastreg_wr.wr; } frd-&gt;frd_fastreg_wr.wr.next = &amp;tx-&gt;tx_wrq[0].wr; will_post = <span class="code-keyword">true</span>; } &lt;snip&gt; The post is here <span class="code-keyword">if</span> (lnet_send_error_simulation(tx-&gt;tx_lntmsg[0], &amp;tx-&gt;tx_hstatus)) rc = -EINVAL; <span class="code-keyword">else</span> #ifdef HAVE_IB_POST_SEND_RECV_CONST rc = ib_post_send(conn-&gt;ibc_cmid-&gt;qp, wr, (<span class="code-keyword">const</span> struct ib_send_wr **)&amp;bad); #<span class="code-keyword">else</span> rc = ib_post_send(conn-&gt;ibc_cmid-&gt;qp, wr, &amp;bad); #endif conn-&gt;ibc_last_send = ktime_get(); <span class="code-keyword">if</span> (rc == 0) <span class="code-keyword">return</span> 0; &lt;-- <span class="code-keyword">return</span> here with the WRs from the mapping complete &lt;snip&gt; At <span class="code-keyword">this</span> point there is a tx with everthing ready to go, BUT &lt;snip&gt; all posts using the tx until is unmapped will send the invalidate and fast reg &lt;snip&gt; and the invalidate has the OLD key forever since nothing has been done to &lt;snip&gt; remember and disable the invalid WRs <span class="code-keyword">for</span> the next post using the tx. } </pre> </div></div> <p>Everytime the <b>kib_tx</b> is used after the first there will be a superflous <b>OLD</b> invalidate followed by an <b>OLD</b> fast req that will change the key to what it currently is. </p> <p>For hfi1 the second invalidate will get a <b>-EINVAL</b> return code because the keys don't match.</p> <p>There are two possible fixes:</p> <ol> <li>Add state to the kib_fast_reg_descriptor that keeps track of if the fast reg WRs have been posted and patch the post logic to only post the WRs if they had not already been posted.</li> <li>Make the rdmavt invalidate code allow the OLD invalidate and fast reg by only comparing the key bits above 7.</li> </ol> <p>I'm about to attach the two patches for 1.   The patch seems to fix the issue and the lnet_selftest works fine.</p> <p>I'm getting ready to test the invalidate patch, but I should point out our current code works with SRP, iSer, and NFS RDMA as is.</p> <p><span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/39531/39531_linux-kernel-test.patch" title="linux-kernel-test.patch attached to LU-14733">linux-kernel-test.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> contains a potential upstream patch.</p> <p>Older versions attached were incorrect.</p> <p><span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/39531/39531_linux-kernel-test.patch" title="linux-kernel-test.patch attached to LU-14733">linux-kernel-test.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> solves the issue, at the expense of extra processing whenever the kib_tx is reused.</p> <p>Mike,</p> <p>I tried the LND change you suggested on my setup using MOFED, it appears to be fine. </p> <p>Please submit the LND patch for review, it will be easier to track with your ownership.</p> <p>Thanks,</p> <p>Serguei.</p> <blockquote> <p>Please submit the LND patch for review, it will be easier to track with your ownership.</p></blockquote> <p>Which one? There are two patches. One is a bug fix in unmap. The other is the fix to insure duplicate WRs are not sent after the initial posts.</p> <p>Mike</p> <p>If you need help getting your gerrit account sorted out then it's best to email me directly rather than using JIRA <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Peter</p> <p>Mike, </p> <p>I was referring to the bug fix in the kiblnd_fmr_pool_unmap. </p> <p>I have no way of testing the upstream patch with OPA. Is it possible that the kiblnd_fmr_pool_unmap fix is sufficient by itself?</p> <p>Thanks,</p> <p>Serguei.</p> <blockquote><p>I have no way of testing the upstream patch with OPA. Is it possible that the kiblnd_fmr_pool_unmap fix is sufficient by itself?</p></blockquote> <p>I tested the rdmavt upstream fix by rebuilding the 8.4 GA kernel from srpm adding the upstream patch.   The unmap fix is not sufficient by itself.  I determined that early on. </p> <p>We either need an upstream patch to get pulled by Jason and point Red Hat to it, or use the second Lustre patch that avoids the superflous WRs that trigger the EINVAL.</p> <p>It seems to me that a Lustre fix might be quicker?</p> <p>Do we need a separate Jira for the map/unmap race fix?</p> <p>I don't think we need a separate ticket for the map/unmap race fix. Peter can correct me if he disagrees.</p> <p>Mike,</p> <p>Thanks for tracking this down.  Do you know why we saw this first with RHEL 8.4, but not RHEL 8.3?</p> <blockquote> <p>Do you know why we saw this first with RHEL 8.4, but not RHEL 8.3?</p></blockquote> <p>Upstream removed FMR as of 5.8 and it looks like the 8.4 RDMA code took that in.</p> <p>8.3 still has the FMR and it appears that this code will prefer it:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #ifdef HAVE_FMR_POOL_API <span class="code-keyword">if</span> (dev-&gt;ibd_dev_caps &amp; IBLND_DEV_CAPS_FMR_ENABLED) rc = kiblnd_alloc_fmr_pool(fps, fpo); <span class="code-keyword">else</span> #endif <span class="code-comment">/* HAVE_FMR_POOL_API */</span> rc = kiblnd_alloc_freg_pool(fps, fpo, dev-&gt;ibd_dev_caps); <span class="code-keyword">if</span> (rc) <span class="code-keyword">goto</span> out_fpo; fpo-&gt;fpo_deadline = ktime_get_seconds() + IBLND_POOL_DEADLINE; fpo-&gt;fpo_owner = fps; *pp_fpo = fpo; <span class="code-keyword">return</span> 0; </pre> </div></div> <p>I don't see any override when both are available?</p> <p>Here are the patches for Lustre:</p> <ol> <li><span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/39552/39552_01-move_null.patch" title="01-move_null.patch attached to LU-14733">01-move_null.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></li> <li><span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/39551/39551_02-post_state.patch" title="02-post_state.patch attached to LU-14733">02-post_state.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></li> </ol> <p>Mike Marciniszyn (mike.marciniszyn@cornelisnetworks.com) uploaded a new patch: <a href="https://review.whamcloud.com/44189" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44189</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Move racy NULL assignment<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 071c8c7faa322be1a42d424ff0d83c3113c80140</p> <p>Mike Marciniszyn (mike.marciniszyn@cornelisnetworks.com) uploaded a new patch: <a href="https://review.whamcloud.com/44190" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44190</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Avoid double posting invalidate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 778962a825eeee5b4754664a6baabf61e982aa38</p> <p>Currently I have unit tested both bulk read and write with opa cards and RHEL8.4.</p> <p>I'm trying to find an MLX card to test with that as well.</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/44189/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44189/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Move racy NULL assignment<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 023113fb8946f3565529e7327fdcd90ab9db3ba3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/44190/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44190/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Avoid double posting invalidate<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5930576791e864529e6ef9b46f3e09cc4b635fc2</p> <p>I saw those two Lustre patches were merged to master.  Has someone been able to test them on MLX to confirm they don't cause new issues there?  Thanks</p> <p>Before Mike pushed them, I tried these patches on my local setup that uses LTS MOFED 4.9 and cx-2 cards. Ran lnet_selftest, didn't see any issues.</p> <blockquote> <p>Before Mike pushed them, I tried these patches on my local setup that uses LTS MOFED 4.9 and cx-2 cards. Ran lnet_selftest, didn't see any issues.</p></blockquote> <p>I will also test on some MLX cards. I will try the stock 8.4 kernel vs. MOFED though.</p> <p>Great, thank you both.</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44216" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44216</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Move racy NULL assignment<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: d33a2dcf34fcea3adaf46d1a806405a9c334adc0</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/44217" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44217</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Avoid double posting invalidate<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 8c70059776f2bdc7a68a27e35bb5dc36763d3dd6</p> <blockquote> <p>05:00.0 Infiniband controller: Mellanox Technologies MT27700 Family <span class="error">&#91;ConnectX-4&#93;</span></p></blockquote> <p>I was able to run lnet_selftest on a vanilla RH 8.4 install:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@ioperf-05 ~]# ./lnet_wrapper_read LST_SESSION = 6780 SESSION: lstread FEATURES: 1 TIMEOUT: 300 FORCE: No 192.168.25.9@o2ib1 are added to session 192.168.25.10@o2ib1 are added to session Test was added successfully bulk_read is running now Capturing statistics for 30 secs [LNet Rates of lfrom] [R] Avg: 21275 RPC/s Min: 21275 RPC/s Max: 21275 RPC/s [W] Avg: 10638 RPC/s Min: 10638 RPC/s Max: 10638 RPC/s [LNet Bandwidth of lfrom] [R] Avg: 10639.30 MiB/s Min: 10639.30 MiB/s Max: 10639.30 MiB/s [W] Avg: 1.62 MiB/s Min: 1.62 MiB/s Max: 1.62 MiB/s [LNet Rates of lto] [R] Avg: 10637 RPC/s Min: 10637 RPC/s Max: 10637 RPC/s [W] Avg: 21275 RPC/s Min: 21275 RPC/s Max: 21275 RPC/s [LNet Bandwidth of lto] [R] Avg: 1.62 MiB/s Min: 1.62 MiB/s Max: 1.62 MiB/s [W] Avg: 10639.10 MiB/s Min: 10639.10 MiB/s Max: 10639.10 MiB/s [LNet Rates of lfrom] [R] Avg: 21294 RPC/s Min: 21294 RPC/s Max: 21294 RPC/s [W] Avg: 10647 RPC/s Min: 10647 RPC/s Max: 10647 RPC/s [LNet Bandwidth of lfrom] [R] Avg: 10647.90 MiB/s Min: 10647.90 MiB/s Max: 10647.90 MiB/s [W] Avg: 1.62 MiB/s Min: 1.62 MiB/s Max: 1.62 MiB/s [LNet Rates of lto] [R] Avg: 10647 RPC/s Min: 10647 RPC/s Max: 10647 RPC/s [W] Avg: 21294 RPC/s Min: 21294 RPC/s Max: 21294 RPC/s [LNet Bandwidth of lto] [R] Avg: 1.62 MiB/s Min: 1.62 MiB/s Max: 1.62 MiB/s [W] Avg: 10647.90 MiB/s Min: 10647.90 MiB/s Max: 10647.90 MiB/s [LNet Rates of lfrom] [R] Avg: 21275 RPC/s Min: 21275 RPC/s Max: 21275 RPC/s [W] Avg: 10637 RPC/s Min: 10637 RPC/s Max: 10637 RPC/s [LNet Bandwidth of lfrom] [R] Avg: 10639.50 MiB/s Min: 10639.50 MiB/s Max: 10639.50 MiB/s [W] Avg: 1.62 MiB/s Min: 1.62 MiB/s Max: 1.62 MiB/s [LNet Rates of lto] [R] Avg: 10637 RPC/s Min: 10637 RPC/s Max: 10637 RPC/s [W] Avg: 21276 RPC/s Min: 21276 RPC/s Max: 21276 RPC/s [LNet Bandwidth of lto] [R] Avg: 1.62 MiB/s Min: 1.62 MiB/s Max: 1.62 MiB/s [W] Avg: 10639.50 MiB/s Min: 10639.50 MiB/s Max: 10639.50 MiB/s [LNet Rates of lfrom] [R] Avg: 21297 RPC/s Min: 21297 RPC/s Max: 21297 RPC/s [W] Avg: 10649 RPC/s Min: 10649 RPC/s Max: 10649 RPC/s [LNet Bandwidth of lfrom] [R] Avg: 10649.22 MiB/s Min: 10649.22 MiB/s Max: 10649.22 MiB/s [W] Avg: 1.62 MiB/s Min: 1.62 MiB/s Max: 1.62 MiB/s [LNet Rates of lto] [R] Avg: 10648 RPC/s Min: 10648 RPC/s Max: 10648 RPC/s [W] Avg: 21294 RPC/s Min: 21294 RPC/s Max: 21294 RPC/s [LNet Bandwidth of lto] [R] Avg: 1.62 MiB/s Min: 1.62 MiB/s Max: 1.62 MiB/s [W] Avg: 10649.22 MiB/s Min: 10649.22 MiB/s Max: 10649.22 MiB/s [LNet Rates of lfrom] [R] Avg: 21304 RPC/s Min: 21304 RPC/s Max: 21304 RPC/s [W] Avg: 10652 RPC/s Min: 10652 RPC/s Max: 10652 RPC/s [LNet Bandwidth of lfrom] [R] Avg: 10653.70 MiB/s Min: 10653.70 MiB/s Max: 10653.70 MiB/s [W] Avg: 1.63 MiB/s Min: 1.63 MiB/s Max: 1.63 MiB/s [LNet Rates of lto] [R] Avg: 10652 RPC/s Min: 10652 RPC/s Max: 10652 RPC/s [W] Avg: 21305 RPC/s Min: 21305 RPC/s Max: 21305 RPC/s [LNet Bandwidth of lto] [R] Avg: 1.63 MiB/s Min: 1.63 MiB/s Max: 1.63 MiB/s [W] Avg: 10653.70 MiB/s Min: 10653.70 MiB/s Max: 10653.70 MiB/s lfrom: Total 0 error nodes in lfrom lto: Total 0 error nodes in lto 1 batch in stopping Batch is stopped session is ended </pre> </div></div> <p>The only issue I ran into was a panic trying to reboot. The servers required a powercyle.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Jul 12 19:09:14 ioperf-06 kernel: reboot D 0 10092 9865 0x00004080 Jul 12 19:09:14 ioperf-06 kernel: Call Trace: Jul 12 19:09:14 ioperf-06 kernel: __schedule+0x2c4/0x700 Jul 12 19:09:14 ioperf-06 kernel: ? __switch_to_asm+0x35/0x70 Jul 12 19:09:14 ioperf-06 kernel: ? __switch_to_asm+0x35/0x70 Jul 12 19:09:14 ioperf-06 kernel: schedule+0x38/0xa0 Jul 12 19:09:14 ioperf-06 kernel: schedule_timeout+0x246/0x2f0 Jul 12 19:09:14 ioperf-06 kernel: ? __switch_to_asm+0x41/0x70 Jul 12 19:09:14 ioperf-06 kernel: ? __switch_to+0x10c/0x480 Jul 12 19:09:14 ioperf-06 kernel: ? __schedule+0x2cc/0x700 Jul 12 19:09:14 ioperf-06 kernel: wait_for_completion+0x97/0x100 Jul 12 19:09:14 ioperf-06 kernel: cma_remove_one+0x23f/0x310 [rdma_cm] Jul 12 19:09:14 ioperf-06 kernel: remove_client_context+0x8b/0xd0 [ib_core] Jul 12 19:09:14 ioperf-06 kernel: disable_device+0x8c/0x130 [ib_core] Jul 12 19:09:14 ioperf-06 kernel: __ib_unregister_device+0x35/0xa0 [ib_core] Jul 12 19:09:14 ioperf-06 kernel: ib_unregister_device+0x21/0x30 [ib_core] Jul 12 19:09:14 ioperf-06 kernel: __mlx5_ib_remove+0x38/0x60 [mlx5_ib] Jul 12 19:09:14 ioperf-06 kernel: mlx5_detach_device+0xb2/0xc0 [mlx5_core] Jul 12 19:09:14 ioperf-06 kernel: mlx5_unload_one+0x80/0x120 [mlx5_core] Jul 12 19:09:14 ioperf-06 kernel: shutdown+0x144/0x1d0 [mlx5_core] Jul 12 19:09:14 ioperf-06 kernel: pci_device_shutdown+0x34/0x60 Jul 12 19:09:14 ioperf-06 kernel: device_shutdown+0x161/0x212 Jul 12 19:09:14 ioperf-06 kernel: kernel_restart+0xe/0x30 Jul 12 19:09:14 ioperf-06 kernel: __do_sys_reboot+0x1d2/0x210 Jul 12 19:09:14 ioperf-06 kernel: ? syscall_trace_enter+0x1d3/0x2c0 Jul 12 19:09:14 ioperf-06 kernel: ? __audit_syscall_exit+0x249/0x2a0 Jul 12 19:09:14 ioperf-06 kernel: do_syscall_64+0x5b/0x1a0 Jul 12 19:09:14 ioperf-06 kernel: entry_SYSCALL_64_after_hwframe+0x65/0xca Jul 12 19:09:14 ioperf-06 kernel: RIP: 0033:0x7f73aa2825f7 Jul 12 19:09:14 ioperf-06 kernel: Code: Unable to access opcode bytes at RIP 0x7f73aa2825cd. Jul 12 19:09:14 ioperf-06 kernel: RSP: 002b:00007ffed2874d28 EFLAGS: 00000246 ORIG_RAX: 00000000000000a9 Jul 12 19:09:14 ioperf-06 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f73aa2825f7 Jul 12 19:09:14 ioperf-06 kernel: RDX: 0000000001234567 RSI: 0000000028121969 RDI: 00000000fee1dead Jul 12 19:09:14 ioperf-06 kernel: RBP: 00007ffed2874d70 R08: 0000000000000002 R09: 0000000000000000 Jul 12 19:09:14 ioperf-06 kernel: R10: 000000000000004b R11: 0000000000000246 R12: 0000000000000001 Jul 12 19:09:14 ioperf-06 kernel: R13: 00000000fffffffe R14: 0000000000000006 R15: 0000000000000000 </pre> </div></div> <p>Gian-Carlo DeFazio (defazio1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/44295" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44295</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Move racy NULL assignment<br/> Project: fs/lustre-release<br/> Branch: b2_14<br/> Current Patch Set: 1<br/> Commit: 8db93a26f7e99f140ccd4a0fd3e35f4e9f71b8ec</p> <p>Gian-Carlo DeFazio (defazio1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/44296" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44296</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Avoid double posting invalidate<br/> Project: fs/lustre-release<br/> Branch: b2_14<br/> Current Patch Set: 1<br/> Commit: 1105bfbfad5002bc673ff568c0720bcebc5d095a</p> <p>Mike, Serguei, this works on our test system</p> <p>Landed for 2.15</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44216/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44216/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Move racy NULL assignment<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 173d60a3274d19bc1d9811b6e1b09aac2b25f221</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44217/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44217/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Avoid double posting invalidate<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 96d7dcf4e773e6026a590e4596ef30ac8a4a5061</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44295/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44295/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Move racy NULL assignment<br/> Project: fs/lustre-release<br/> Branch: b2_14<br/> Current Patch Set: <br/> Commit: 380be07fcca1f76564d1f29e58f2d8d5f8f530c8</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44296/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44296/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14733" title="brw_bulk_ready() BRW bulk READ failed for RPC from 12345-192.168.128.126@o2ib18: -103" class="issue-link" data-issue-key="LU-14733"><del>LU-14733</del></a> o2iblnd: Avoid double posting invalidate<br/> Project: fs/lustre-release<br/> Branch: b2_14<br/> Current Patch Set: <br/> Commit: 29da7cba3e7b3461d895010c7f7284b9649aba52</p> Duplicate LU-13976 Related LU-15116 Development Rank 1|i01w2n: Rank (Obsolete) 9223372036854775807 Severity