https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11257 Lustre <p>It looks like the latest kernel update from CentOS/RedHat prevents LNet to work on Infiniband interfaces (mlx5).</p> <h3><a name="Symptoms"></a>Symptoms</h3> <p>No LNet communication, self-ping doesn't work:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # lctl list_nids 10.9.101.60@o2ib4 # lctl ping 10.9.101.60@o2ib4 failed to ping 10.9.101.60@o2ib4: Input/output error</pre> </div></div> <p>Communicating with other nodes is impossible, as is mounting filesystems.<br/> The exact same node with the exact same configuration works flawlessly with kernel <tt>3.10.0-862.9.1.el7.x86_64</tt></p> <h3><a name="%C2%A0Versions"></a> Versions</h3> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # uname -r 3.10.0-862.11.6.el7.x86_64 # cat /sys/fs/lustre/version 2.10.4</pre> </div></div> <h3><a name="HW"></a>HW</h3> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ibstat CA 'mlx5_0' CA type: MT4115 Number of ports: 1 Firmware version: 12.21.3012 Hardware version: 0 Node GUID: 0x7cfe900300268c04 System image GUID: 0x7cfe900300268c04 Port 1: State: Active Physical state: LinkUp Rate: 100 Base lid: 72 LMC: 0 SM lid: 6 Capability mask: 0x2651e848 Port GUID: 0x7cfe900300268c04 Link layer: InfiniBand</pre> </div></div> <p> </p> <h3><a name="Kernellogs"></a>Kernel logs</h3> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [ 1185.337098] LNetError: 22109:0:(o2iblnd_cb.c:2513:kiblnd_passive_connect()) Can't accept 10.9.101.60@o2ib4: -22 [ 1185.348376] LNet: 22109:0:(o2iblnd_cb.c:2212:kiblnd_reject()) Error -22 sending reject [ 1185.357473] LNetError: 22109:0:(o2iblnd_cb.c:2721:kiblnd_rejected()) 10.9.101.60@o2ib4 rejected: consumer defined fatal error</pre> </div></div> CentOS 7.5, x86_64 LU-11257

RHEL/CentOS 3.10.0-862.11.6.el7.x86_64 kernel breaks LNet

Bug Major Resolved Duplicate Peter Jones Stanford Research Computing Center Thu, 16 Aug 2018 03:23:53 +0000 Fri, 23 Aug 2019 13:28:02 +0000 Fri, 23 Aug 2019 13:28:01 +0000 Lustre 2.10.4 0 11 <p>A more detailed changelog about that kernel is at <a href="https://access.redhat.com/downloads/content/rhel---7/x86_64/2456/kernel/3.10.0-862.11.6.el7/x86_64/fd431d51/package-changelog," class="external-link" target="_blank" rel="nofollow noopener">https://access.redhat.com/downloads/content/rhel---7/x86_64/2456/kernel/3.10.0-862.11.6.el7/x86_64/fd431d51/package-changelog,</a> if that's of any help.</p> <p>we see the same thing on our OPA network.</p> <p>ksocklnd reportedly seems ok with this kernel on our TCP networks (in VMs mostly), so I suspect it's ko2iblnd related.</p> <p>below is syslog from<br/> john98 # lctl ping warble@o2ib44</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Aug 17 01:58:10 john98 kernel: LNet: HW NUMA nodes: 2, HW CPU cores: 36, npartitions: 2 Aug 17 01:58:10 john98 kernel: alg: No test for adler32 (adler32-zlib) Aug 17 01:58:11 john98 kernel: Lustre: Lustre: Build Version: 2.10.4 Aug 17 01:58:11 john98 kernel: LNet: Using FMR for registration Aug 17 01:58:11 john98 kernel: LNet: Added LNI 192.168.44.198@o2ib44 [128/2048/0/180] Aug 17 01:58:36 warble1 kernel: LNetError: 103:0:(o2iblnd_cb.c:3061:kiblnd_cm_callback()) 192.168.44.198@o2ib44: REJECTED 28 Aug 17 01:58:36 warble1 kernel: LNetError: 103:0:(o2iblnd_cb.c:3061:kiblnd_cm_callback()) Skipped 3 previous similar messages Aug 17 02:06:08 john98 kernel: LNetError: 204:0:(o2iblnd_cb.c:2513:kiblnd_passive_connect()) Can't accept 192.168.44.198@o2ib44: -22 Aug 17 02:06:08 john98 kernel: LNet: 204:0:(o2iblnd_cb.c:2212:kiblnd_reject()) Error -22 sending reject Aug 17 02:06:08 john98 kernel: LNetError: 204:0:(o2iblnd_cb.c:2721:kiblnd_rejected()) 192.168.44.198@o2ib44 rejected: consumer defined fatal error </pre> </div></div> <p>2.10.4 was dkms rebuilt for this kernel.</p> <p>cheers,<br/> robin</p> <p>actually, ib_send_rw and ibv_rc_pingpong don't seem to work on this new kernel, so I suspect a RHEL have broken all IB RDMA?</p> <p>do they work for you?</p> <p>IPoIB works ok.</p> <p>cheers,<br/> robin</p> <p>Good observation, indeed: perf tests such as ib_{read,send}_bw  and ibv_rc_pingpong fail with errors like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Failed to modify QP to RTR Couldn't connect to remote QP </pre> </div></div> <p>or</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Failed to modify QP 386 to RTR Unable to Connect the HCA's through the link </pre> </div></div> <p>I submitted RHEL bug #1618452 to report the issue:<br/> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1618452" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1618452</a></p> <p>Which seem to be marked "private" by the Redhat bugzilla, without any way to mark it "public" on my end <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I think that you have to request for them to open it up.</p> <p>RHEL's reply:</p> <blockquote><p><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1618452" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1618452</a></p> <p>— Comment #3 from Don Dutile &lt;<span class="nobr"><a href="mailto:ddutile@redhat.com" class="external-link" target="_blank" rel="nofollow noopener">ddutile@redhat.com^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span>&gt; — <br/> Already reported and being actively fixed.</p> <p>Cannot make this public, as the patch that caused it was due to embargo'd <br/> security fix.</p> <p>This issue has highest priority for resolution.<br/> Revert to 3.10.0-862.11.5.el7 in the mean time.</p> <p>This bug has been marked as a duplicate of bug 1616346</p></blockquote> <p> </p> <p> </p> <p>Thanks for the info!</p> <p>Still no update from Red Hat. </p> <p>We're getting more info via The Register:<br/> <a href="https://www.theregister.co.uk/2018/08/21/fix_for_julys_spectrelike_bug_is_breaking_some_supers/" class="external-link" target="_blank" rel="nofollow noopener">https://www.theregister.co.uk/2018/08/21/fix_for_julys_spectrelike_bug_is_breaking_some_supers/</a></p> <blockquote><p>“The problem will be fixed in kernel-3.10.0-862.13.1 which is currently being reviewed by Red Hat Enterprise Linux Engineering.”</p></blockquote> <p>But no ETA yet.</p> <p>hmm. comments attached to that article point to a fix in centos - potentially just a misplaced semi-colon. but OTOH the centos bug seems to be talking about IPoIB and that works fine. perhaps the fix is right and the bug report is wrong?</p> <p>cheers,<br/> robin</p> <p>I wish I would have looked here first when digging into the same thing instead of wasting a day trying to figure out the culprit.  For now I've opened another redhat bug since I didn't come across anything when searching their bugzilla.:</p> <p><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1625620" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1625620</a></p> <p>FYI <a href="https://bugs.centos.org/view.php?id=15193" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.centos.org/view.php?id=15193</a></p> <p>yeah, we gave up waiting and just built our own ib_core.ko module with the 1-character patch from centos.<br/> works fine now.</p> <p>cheers,<br/> robin</p> <p>Kernel 3.10.0-862.14 has been released, which fixes the issue:</p> <p> <a href="https://access.redhat.com/downloads/content/rhel---7/x86_64/2456/kernel/3.10.0-862.14.4.el7/x86_64/fd431d51/package" class="external-link" target="_blank" rel="nofollow noopener">https://access.redhat.com/downloads/content/rhel---7/x86_64/2456/kernel/3.10.0-862.14.4.el7/x86_64/fd431d51/package</a></p> <p>The fix has also been noted in the Centos bug report; <a href="https://bugs.centos.org/view.php?id=15193." class="external-link" target="_blank" rel="nofollow noopener">https://bugs.centos.org/view.php?id=15193.</a>  The update .rpm isn't available in Centos mirrors yet though.</p> <p> </p> <p>the kernel update to 3.10.0-862.14.4 is now available on Centos mirrors</p> <p> </p> <p>RHEL 7.5 kernel update to 3.10.0-862.14.4.el7 is tracked in <a href="https://jira.whamcloud.com/browse/LU-11448" title="kernel update [RHEL7.5 3.10.0-862.14.4.el7]" class="issue-link" data-issue-key="LU-11448"><del>LU-11448</del></a>.</p> <p>It seems like this was fixed in the next RHEL/CentOS update</p> Duplicate LU-11261 Related LU-11253 Development Rank 1|i000tj: Rank (Obsolete) 9223372036854775807 Severity