https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11348 Lustre <p> </p> <p>When building Lustre 2.10.4 with Mellanox OFED stack, got errors:</p> <p>configure: error: can't compile with OpenIB gen2 headers</p> <p>(please see attached file for more details)</p> [<a href='mailto:root@wombat-oss-21-2'>root@wombat-oss-21-2</a> ~]# uname -a <br/> Linux wombat-oss-21-2.local 2.6.32-573.7.1.el6.x86_64 #1 SMP Tue Sep 22 22:00:00 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux <br/> <br/> [<a href='mailto:root@wombat-oss-21-2'>root@wombat-oss-21-2</a> ~]# rpm -aq | egrep &quot;spl|zfs&quot; <br/> spl-0.7.9-1.el6.x86_64 <br/> zfs-0.7.9-1.el6.x86_64 <br/> kmod-spl-0.7.9-1.el6.x86_64 <br/> kmod-spl-devel-0.7.9-1.el6.x86_64 <br/> kmod-zfs-0.7.9-1.el6.x86_64 <br/> zfs-dracut-0.7.9-1.el6.x86_64 <br/> kmod-zfs-devel-0.7.9-1.el6.x86_64 <br/> lustre-osd-zfs-mount-2.10.4-1.el6.x86_64 <br/> libzfs2-0.7.9-1.el6.x86_64 <br/> libzfs2-devel-0.7.9-1.el6.x86_64 <br/> kmod-lustre-osd-zfs-2.10.4-1.el6.x86_64 <br/> LU-11348

Lustre 2.10.4 failed to build with MLNX_OFED_LINUX-4.4-1.0.0.0

Bug Major Resolved Not a Bug Minh Diep Haisong Cai build Thu, 6 Sep 2018 05:08:04 +0000 Fri, 14 Sep 2018 14:02:43 +0000 Fri, 14 Sep 2018 14:02:43 +0000 Lustre 2.10.4 0 3 <p>Minh</p> <p>Could you please advise</p> <p>Peter</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=haisong" class="user-hover" rel="haisong">haisong</a>,</p> <p>I think you need --with-o2ib=/usr/src/ofa_kernel/default</p> <p>Hi Minh,</p> <p>Good to see you!</p> <p><tt>Well, I tried it, with various alternatives pathe besides /usr/src/ofa_kernel/default, such as mlnx-ofa_kernel-4.4.</tt><tt>The same errors came out.</tt><tt>I also noticed <a href="https://jira.whamcloud.com/browse/LU-6790" title="can&#39;t compile with OpenIB gen2 headers under /usr/src/compat-rdma" class="issue-link" data-issue-key="LU-6790"><del>LU-6790</del></a> mentioning CentOS6.6 wasn't being support?</tt><tt>More system info below:</tt></p> <ol> <li>ofed_info -s<br/> MLNX_OFED_LINUX-4.4-1.0.0.0:</li> </ol> <ol> <li>ls -1 /usr/src/<br/> debug<br/> kernels<br/> linux-2.6<br/> mlnx-ofa_kernel<br/> mlnx-ofa_kernel-4.4<br/> ofa_kernel<br/> ofa_kernel-4.4<br/> spl-0.6.4.2<br/> spl-0.7.9<br/> zfs-0.6.4.2<br/> zfs-0.7.9</li> </ol> <p> </p> <p><tt>Thanks,</tt><tt>Haisong</tt></p> <p>Hey,</p> <p>yes, el6.6 is quite old</p> <p> </p> <p>Hi Minh,</p> <p>I tried to compile server 2.10.4 and 2.10.5 on a CentOS 6.10, ZFS 0.7.9 machine.</p> <p>Lustre server rpms built. But when try to install, I get errors - see attachment.<span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31016/31016_lustre-installation-errors" title="lustre-installation-errors attached to LU-11348">lustre-installation-errors^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p> </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=haisong" class="user-hover" rel="haisong">haisong</a><br/> Below was my recommendation on el7 but also applies to el6<br/> The dependencies errors are due to different in build process between lustre and MOFED. Here is one way that works</p> <p>1. download the MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.5-x86_64.tgz; unzip and cd MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.5-x86_64<br/> 2. you would use mlnx_add_kernel_support.sh since the kernel is newer than MOFED support.<br/> Run mlnx_add_kernel_support.sh --kmp (enable kmp support since it's disabled by default)<br/> 3. once a build is done, there is a new tarball in /tmp/MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.5-x86_64-ext.tgz<br/> 4. unzip MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.5-x86_64-ext.tgz and go to MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.5-x86_64-ext/RPMS<br/> 5. install these rpms (others too if you think you'll need them)</p> <p>yum localinstall mlnx-ofa_kernel-4.4-OFED.4.4.2.0.7.1.gee7aa0e.rhel7u5.x86_64.rpm kmod-mlnx-ofa_kernel-4.4-OFED.4.4.2.0.7.1.gee7aa0e.201808301559.rhel7u5.x86_64.rpm mlnx-ofa_kernel-devel-4.4-OFED.4.4.2.0.7.1.gee7aa0e.rhel7u5.x86_64.rpm</p> <p>Now you can build lustre<br/> ./configure --with-obib=/usr/src/ofa_kernel/default &amp;&amp; make &amp;&amp; make rpms</p> <p>Please let me know the result</p> <p>^Minh,</p> <p>^{I followed suggested steps, but it f}^{ailed still.}</p> <p>^{Including compilation log and installation log here.}</p> <p>^Thanks,</p> <p>^Haisong<span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/31025/31025_lustre_compiling_installing_log" title="lustre_compiling_installing_log attached to LU-11348">lustre_compiling_installing_log^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>Minh,</p> <p>Are the following compatible:</p> <p><tt>CentOS6.10 (kernel 2.6.32-754.3.5.el6.x86_64)</tt></p> <p>ZFS 0.7.9</p> <p>Lustre server 2.10.4</p> <p>Mellanox MLNX_OFED_LINUX-4.4-1.0.0.0-rhel6.10-x86_64</p> <p>I still can't build Lustre server.</p> <p>Thanks,</p> <p>Haisong</p> <p>We offer only support for Lustre clients for RHEL 6.x on the LTS branch.</p> <p>Thank you for the update.</p> <p>However in my earlier ticket, EDU-91, engineer suggested that we upgrade Lustre server from FE2.7.x to 2.10.x for that particular MDS bug.</p> <p>Now we are preparing for such upgrade, LTS is no longer supported? Here I am assuming LTS referring to all 2.10.x (according to the release note here <a href="http://lustre.org/lustre-2-10-0-released)" class="external-link" target="_blank" rel="nofollow noopener">http://lustre.org/lustre-2-10-0-released)</a></p> <p> </p> <p>Thanks,</p> <p>Haisong</p> <p>I should also add that, on Aug 23, I used a CentOS6.7 server + ZFS 0.7.9, with Lustre 2.10.4 (git checkout v2_10_4) and successfully built and installed the server.</p> <p>Now that process can't be repeated any more.</p> <p>Haisong</p> <p>Haisong</p> <p>Sorry to not have been clear enough - the 2.10.x LTS branch is absolutely supported, but only for Linux distributions as outlined in the support matrix - <a href="https://wiki.whamcloud.com/display/PUB/Lustre+Support+Matrix" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.whamcloud.com/display/PUB/Lustre+Support+Matrix</a> . As you can see, we only offer support for RHEL 7.x servers at present.</p> <p>Reviewing the exchange on the EDU-91 ticket it is clear to me that the enquiry as to whether you were planning to upgrade to 2.10.x assumed that you realized that the implication of this would also mean to upgrade the Linux distribution on the storage servers accordingly.</p> <p>We don't try and break support for older Linux distributions so they can often stay working long after they have been dropped from the combinations we build and test. I am more surprised that it is only so recently RHEL 6.x servers stopped working than to learn that they do not work. It is some years since we have been actively supporting RHEL 6.x servers on master (they were deprecated for the Lustre 2.8 release cycle)</p> <p>Given your requirements I see two options:</p> <p>1) Upgrade your storage servers distro to RHEL 7.5 and keep your existing kernel on your clients, using 2.10.5</p> <p>2) Keep your storage servers on 2.7 FE (the last release to support RHEL 6.x servers) and see whether it is possible to port the <a href="https://jira.whamcloud.com/browse/LU-7199" title="Null pointer dereference in old_init_ucred" class="issue-link" data-issue-key="LU-7199"><del>LU-7199</del></a> fix back to that version</p> <p>Personally, I think that option #1 is the better approach. TBH I am surprised that you are still using RHEL 6.x for a Lustre on ZFS deployment - a study conducted back when Rick Wagner was as SDSC had showed that there were performance advantages of using RHEL 7.x over RHEL 6.x for Lustre on ZFS. Many sites will take this approach and treat the storage servers as a "black box" rather than trying to have a single kernel version used across both Lustre servers and clients.</p> <p>Peter</p> <p>Minh,</p> <p>I followed your suggestion and built 2.10.4 client successfuly on CentOS6.10.</p> <p>The procesures is very close to what I used for building server, minus ZFS</p> <p>command summary:</p> <p>tar zxvf /share/apps/src/MLNX_OFED_LINUX-4.4-1.0.0.0-rhel6.10-x86_64.tgz</p> <p>cd MLNX_OFED_LINUX-4.4-1.0.0.0-rhel6.10-x86_64<br/> ./mlnx_add_kernel_support.sh -m /tmp/MLNX_OFED_LINUX-4.4-1.0.0.0-rhel6.10-x86_64</p> <p>tar xvf /tmp/MLNX_OFED_LINUX-4.4-1.0.0.0-rhel6.10-x86_64-ext.tgz<br/> cd MLNX_OFED_LINUX-4.4-1.0.0.0-rhel6.10-x86_64-ext<br/> ./mlnxofedinstall</p> <p><span class="error">&#91;root@badger-oss-4-1 MLNX_OFED_LINUX-4.4-1.0.0.0-rhel6.10-x86_64-ext&#93;</span># ofed_info -s<br/> MLNX_OFED_LINUX-4.4-1.0.0.0:</p> <ol> <li> <ol> <li> <ol> <li>Lustre client build<br/> cd /tmp/<br/> tar zxvf /share/apps/src/lustre-release_git.tgz<br/> cd /tmp/lustre-release<br/> sh ./autogen.sh<br/> ./configure --disable-server --with-obib=/usr/src/ofa_kernel/default &amp;&amp; make &amp;&amp; make rpms</li> </ol> </li> </ol> </li> </ol> <p> </p> <p>I also saved logs. Let me know if you like them too.</p> <p>Minh,</p> <p><tt>After further troubleshooting, I found my problem to the original issue - Lustre 2.10.4 failed to build on CentOS6.10.</tt></p> <p><tt>It was really a run-time issue, with Lustre build and install process, I believe.</tt></p> <p><tt>It turns out the processes requires Mellanox card to be presented and driver loaded in kernel space first. The machine I used to build Lustre server didn't have the Mellanox card (it was there before) at the time.</tt></p> <p><tt>So even Lustre 2.10.X isn't officially supported in CentOS6.* anymore, one can still build it on the platform and use it.</tt></p> <p><tt>You can close this ticket.</tt></p> <p><tt>Thanks for all the helps,</tt></p> <p><tt>Haisong</tt></p> <p>I find it's hard to believe that we need a card to build. I have built lustre on VM all the time without the card. Perhaps you need to use --with-linux option?</p> <p>-Minh</p> <p> </p> <p>I used --with-linux option option.</p> <p>With Mellanox card, this time I only run with</p> <p>./configure --enable-server --disable-ldiskfs --with-o2ib=yes</p> <p>This is the option I usually use.</p> <p>Thanks,</p> <p>Haisong</p> <p>if you use --with-o2ib, the config will assume that you want to build with whatever IB installed on the system which mean you need to have IB card.<br/> if you don't have IB driver or card, use --with-o2ib=/usr/src/ofa_kernel/default</p> <p>In the course of my week long troubleshooting, I tried everything I could think of, including the one you suggested here.</p> <p>At this point, I am certaint that with the IB card in the server, all my configuration options will likely success. In that regard, The configure part of the compilation step is pretty smart about figuring out proper configuration parameters.</p> <p>Haisong</p> <p>Thanks</p> Development Epic server Epic/Theme Mellaonx Rank 1|i001xr: Rank (Obsolete) 9223372036854775807 Severity