https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15534 Lustre <p>Since installing TOSS 4.3-X (a RHEL 8.5 derivative) on lustre servers, we've had an issue with lnet.</p> <p>We are unable to successfully lnetctl ping between nodes when using infiniband as the underlying network. There is no indication of problems with IB:</p> <ul> <li>"ping" (the unix utility) between the two nodes via IPoIB is successful, in either direction</li> <li>ib_write_bw between the two nodes via the IB network is successful, in either direction</li> </ul> <p>When LNet starts, it begins reporting the following on the console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LNetError: 24429:0:(lib-move.c:3756:lnet_handle_recovery_reply()) peer NI (172.19.1.54@o2ib100) recovery failed with -110 </pre> </div></div> <p>Eventually, we see the following on the console:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>INFO: task kworker/u128:2:5350 blocked for more than 120 seconds.       Tainted: P           OE    --------- -t - 4.18.0-348.7.1.1toss.t4.x86_64 #1 "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:kworker/u128:2  state:D stack:    0 pid: 5350 ppid:     2 flags:0x80004080 Workqueue: rdma_cm cma_work_handler [rdma_cm] Call Trace: __schedule+0x2c0/0x770 schedule+0x4c/0xc0 schedule_preempt_disabled+0x11/0x20 __mutex_lock.isra.6+0x343/0x550 rdma_connect+0x1e/0x40 [rdma_cm] kiblnd_cm_callback+0x14ee/0x2230 [ko2iblnd] ? __switch_to_asm+0x41/0x70 cma_cm_event_handler+0x25/0xf0 [rdma_cm] cma_work_handler+0x5a/0xb0 [rdma_cm] process_one_work+0x1ae/0x3a0 worker_thread+0x3c/0x3c0 ? create_worker+0x1a0/0x1a0 kthread+0x12f/0x150 ? kthread_flush_work_fn+0x10/0x10 ret_from_fork+0x1f/0x40 </pre> </div></div> TOSS 4.3 (based on RHEL 8.5) <br/> 4.18.0-348.7.1.1toss.t4.x86_64 <br/> lustre 2.14.0_10.llnl <br/> LU-15534

failed to ping 172.19.1.27@o2ib100: Input/output error

Bug Major Closed Fixed Serguei Smirnov Gian-Carlo Defazio llnl Tue, 8 Feb 2022 00:33:55 +0000 Fri, 11 Feb 2022 23:27:06 +0000 Thu, 10 Feb 2022 23:30:33 +0000 Lustre 2.14.0 0 4 <p>For my notes the local ticket is at <a href="https://lc.llnl.gov/jira/browse/TOSS-5521" class="external-link" target="_blank" rel="nofollow noopener">https://lc.llnl.gov/jira/browse/TOSS-5521</a></p> <p>So far we've seen this issue only with the RHEL 8.5 kernel and lustre 2.14.</p> <p>The previous version of TOSS, TOSS 4.2-4 is based on RHEL 8.4 and doesn't have this issue. We also haven't seen it on any TOSS 3 systems which are based on RHEL 7.X and running lustre 2.12 or 2.10.</p> <p>This was first noticed on our new storage hardware, which includes the garter cluster.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@garter1:~]# ibstat CA 'mlx5_0'         CA type: MT4119         Number of ports: 1         Firmware version: 16.31.1014         Hardware version: 0         Node GUID: 0x0c42a103008ee90a         System image GUID: 0x0c42a103008ee90a         Port 1:                 State: Active                 Physical state: LinkUp                 Rate: 100                 Base lid: 391                 LMC: 0                 SM lid: 363                 Capability mask: 0x2659e848                 Port GUID: 0x0c42a103008ee90a                 Link layer: InfiniBand </pre> </div></div> <p>The subnet manager listed, orelic1, is correct</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@garter1:~]# ibnetdiscover | grep "lid 363" [2]     "H-506b4b0300da6764"[1](506b4b0300da6764)               # "orelic1 mlx5_0" lid 363 4xEDR [1](506b4b0300da6764)   "S-248a0703006d13c0"[2]         # lid 363 lmc 0 "SwitchIB Mellanox Technologies" lid 352 4xEDR </pre> </div></div> <p>The issue is also preset on the boa cluster, which has the same hardware as garter</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@boai:defazio1]# ibstat CA 'mlx5_0'         CA type: MT4119         Number of ports: 1         Firmware version: 16.31.1014         Hardware version: 0         Node GUID: 0x0c42a10300dace36         System image GUID: 0x0c42a10300dace36         Port 1:                 State: Active                 Physical state: LinkUp                 Rate: 100                 Base lid: 228                 LMC: 0                 SM lid: 5                 Capability mask: 0x2659e848                 Port GUID: 0x0c42a10300dace36                 Link layer: InfiniBand </pre> </div></div> <p>It also has the correct subnet manager, zrelic1</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@boai:defazio1]# ibnetdiscover | grep "lid 5 " [5]     "H-7cfe9003000f382e"[1](7cfe9003000f382e)               # "zrelic1 mlx5_0" lid 5 4xEDR [1](7cfe9003000f382e)   "S-7cfe900300b67590"[5]         # lid 5 lmc 0 "SwitchIB Mellanox Technologies" lid 23 4xEDR </pre> </div></div> <p>An older cluster, slag, has the same issue as well.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@slag3:~]# ibstat CA 'mlx5_0'         CA type: MT4115         Number of ports: 1         Firmware version: 12.28.2006         Hardware version: 0         Node GUID: 0x506b4b0300c23712         System image GUID: 0x506b4b0300c23712         Port 1:                 State: Active                 Physical state: LinkUp                 Rate: 100                 Base lid: 359                 LMC: 0                 SM lid: 363                 Capability mask: 0x2659e848                 Port GUID: 0x506b4b0300c23712                 Link layer: InfiniBand </pre> </div></div> <p>We do <em>not</em> see this issue with <br/> RHEL 8.5<br/> kernel 4.18.0-348.7.1.1toss.t4.x86_64<br/> lustre-2.12.8_1.llnl-1.t4.x86_64</p> <p>Serguei</p> <p>Could you please assist with this one?</p> <p>Peter</p> <p>Hi,</p> <p>Could you please provide net debug for the failing ping test?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lctl set_param debug=+net &lt;--- run test ---&gt; lctl dk &gt; log.txt lctl set_param debug=-net</pre> </div></div> <p>Also, could you please provide the configuration script log?</p> <p>Thanks,</p> <p>Serguei.</p> <p>Hi Serguei,</p> <p>I've uploaded some files.</p> <p>2.14.0_10.llnl-x86_64-build.log.gz is the full build log and 2.14.0_10.llnl-x86_64-config.log.gz is from the same build but removes all but the configure portion.</p> <p>I did a lnetctl ping from garter5 (172.19.1.137@o2ib100) to garter6 (172.19.1.138@o2ib100) and included the debug logs for both in garter5_ping-send_2022-02-08_10-53-41 and garter6_ping-receive_2022-02-08_10-53-48.</p> <p>This looks very similar to <a href="https://jira.whamcloud.com/browse/LU-14488" title="Support rdma_connect_locked()" class="issue-link" data-issue-key="LU-14488"><del>LU-14488</del></a> the fix for which appears in 2.12.7. </p> <p>Which MOFED version are you using?</p> <p>Thanks,</p> <p>Serguei</p> <p>We are using OFED.</p> <p> </p> <p><a href="https://jira.whamcloud.com/browse/LU-14488" title="Support rdma_connect_locked()" class="issue-link" data-issue-key="LU-14488"><del>LU-14488</del></a> looks promising.</p> <p>Looking at the source for the 2 kernels, I do not see rdma_connect_locked() in the 4.18.0-305.19.1.el8_4 kernel used to build TOSS 4.2-4, but I do see it in the 4.18.0-348.2.1.el8_5 kernel used to build TOSS 4.3-1.</p> <p>Applying  <a href="https://jira.whamcloud.com/browse/LU-14488" title="Support rdma_connect_locked()" class="issue-link" data-issue-key="LU-14488"><del>LU-14488</del></a> to our local 2.14 branch solved the issue. It looks like it was <a href="https://jira.whamcloud.com/browse/LU-14488" title="Support rdma_connect_locked()" class="issue-link" data-issue-key="LU-14488"><del>LU-14488</del></a></p> <p>Thanks!</p> <p> </p> <p>The issue was fixed by an existing patch that was landed for 2.15.</p> <p>Our local 2.12 branch already had the b2_12 backport of the patch and never experienced this issue.</p> Development Rank 1|i02hkn: Rank (Obsolete) 9223372036854775807 Severity