https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8693 Lustre <p>Moving to our production Power8 system running an MOFED stack we are seeing a new IB error in the ko2iblnd that wasn't encountered before.</p> <p>[ 170.597561] mlx5_warn:mlx5_0:dump_cqe:257:(pid 8738): dump error cqe<br/> [ 170.597620] mlx5_warn:mlx5_0:dump_cqe:257:(pid 8714): dump error cqe<br/> [ 170.597622] 00000000 00000000 00000000 00000000<br/> [ 170.597623] 00000000 00000000 00000000 00000000<br/> [ 170.597625] 00000000 00000000 00000000 00000000<br/> [ 170.597626] 00000000 08007806 25000039 0642b3d2<br/> [ 170.597651] LNet: 8714:0:(o2iblnd_cb.c:3433:kiblnd_complete()) FastReg failed: 6<br/> [ 170.597728] LNet: 8713:0:(o2iblnd_cb.c:3444:kiblnd_complete()) RDMA (tx: c000003c6a78c5a8) failed: 5<br/> [ 170.598355] 00000000 00000000 00000000 00000000<br/> [ 170.598403] 00000000 00000000 00000000 00000000<br/> [ 170.599245] powernv-cpufreq: CPU 104 on Chip 1 has Pmax restored to 0<br/> [ 170.599647] LNet: 8714:0:(o2iblnd_cb.c:990:kiblnd_tx_complete()) Tx -&gt; 10.39.232.11@o2ib6 cookie 0x63e sending 1 waiting 0: failed 5<br/> [ 170.599651] LNet: 8714:0:(o2iblnd_cb.c:990:kiblnd_tx_complete()) Skipped 2 previous similar messages<br/> [ 170.599654] LNet: 8713:0:(o2iblnd_cb.c:1934:kiblnd_close_conn_locked()) Closing conn to 10.39.232.11@o2ib6: error -5(waiting)<br/> [ 170.599669] LustreError: 8714:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc c000003c62cf5c00<br/> [ 170.599675] Lustre: 8896:0:(client.c:2063:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: <span class="error">&#91;sent 1476124274/real 1476124274&#93;</span> req@c000003c4e340000 x1547828424878916/t0(0) o4-&gt;atlastds-OST0035-osc-c000001fc5b75000@10.36.226.69@o2ib:6/4 lens 608/448 e 0 to 1 dl 1476124841 ref 2 fl Rpc:X/0/ffffffff rc 0/-1<br/> [ 170.599681] Lustre: atlastds-OST0035-osc-c000001fc5b75000: Connection to atlastds-OST0035 (at 10.36.226.69@o2ib) was lost; in progress operations using this service will wait for recovery to complete<br/> [ 170.611219] 00000000 00000000 00000000 00000000<br/> [ 170.612270] 00000000 08007806 2500003a 06789cd2<br/> [ 170.613866] LustreError: 8737:0:(events.c:201:client_bulk_callback()) event type 1, status -5, desc c000001fb98c0400</p> Power8 running RHEL with a MOFED 3.3 stack. LU-8693

ko2iblnd recieving IB_WC_MW_BIND_ERR errors.

Bug Critical Resolved Duplicate Amir Shehata James A Simmons Tue, 11 Oct 2016 15:28:03 +0000 Thu, 12 Oct 2017 01:20:08 +0000 Tue, 31 Jan 2017 18:00:02 +0000 Lustre 2.8.0 Lustre 2.9.0 0 9 <p>Doug</p> <p>Could you please advise on this one?</p> <p>Peter</p> <p>James, do you know if this is using FastReg or the older FMR?</p> <p>FastReg</p> <p>Is this only happening with Power8 to/from x86? </p> <p>That all we have.</p> <p>It would be very useful to know under what conditions MOFED returns this error. Without access to the MOFED source or the firmware source (if the error is generated by firmware), I cannot determine that.</p> <p>Do you have a support ticket opened with Mellanox for this? If they can provide us with a list of conditions which generate this error, we would have something to work with to debug what we are doing wrong in o2iblnd.</p> <p>The only reference I can find to IB_WC_MW_BIND_ERR in the upstream OFED code is in Linux/drivers/infiniband/hw/mlx5/cq.c, routine: mlx5_handle_error_cqe():</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... switch (cqe-&gt;syndrome) { ... case MLX5_CQE_SYNDROME_MW_BIND_ERR: wc-&gt;status = IB_WC_MW_BIND_ERR; break; ... </pre> </div></div> <p>I cannot find any other reference to MLX5_CQE_SYNDROME_MW_BIND_ERR so I am assuming this comes from the MLX5 driver or firmware.</p> <p>Does OFED reproduce this error?</p> <p>That's a good question. James? Have you tried the upstream OFED for this?</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a>,<br/> Any thoughts on Doug and Mahmoud's OFED query?</p> <p>We only use OFED 3.12 in our production systems. Also for our Cray systems we don't enable map_on_demand so we don't see any problems.</p> <p>I believe this bug is addressed by the fix to <a href="https://jira.whamcloud.com/browse/LU-8752" title="mlx5_warn:mlx5_0:dump_cqe:257:" class="issue-link" data-issue-key="LU-8752"><del>LU-8752</del></a>.</p> Duplicate LU-8752 Related LU-6387 Development Rank 1|hzyraf: Rank (Obsolete) 9223372036854775807 Severity