[LU-6790] can't compile with OpenIB gen2 headers under /usr/src/compat-rdma

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.7.0
• Fix Version/s: Lustre 2.8.0
• Type: Bug
• Priority: Minor
• Reporter: Marcin Dulak
• Assignee: WC Triage
• Resolution: Fixed

Environment

1. uname -r
   2.6.32-504.8.1.el6_lustre.x86_64

1. rpm -qi kernel-2.6.32-504.8.1.el6_lustre.x86_64
   ...
   Release : 504.8.1.el6_lustre Build Date: Fri 27 Feb 2015 08:21:49 AM CET
   Install Date: Thu 02 Jul 2015 01:10:45 AM CEST Build Host: onyx-7-sdf1-el6-x8664.onyx.hpdd.intel.com
   Group : System Environment/Kernel Source RPM: kernel-2.6.32-504.8.1.el6_lustre.src.rpm
   Size : 130042766 License: GPLv2
   ...

• Severity: 3

Description

Building lustre against IntelIB-Basic.RHEL6-x86_64.7.3.1.0.12.tgz
fails:

1. wget "https://build.hpdd.intel.com/job/lustre-master/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/lastSuccessfulBuild/artifact/artifacts/SRPMS/lustre-2.7.55-2.6.32_431.29.2.el6_lustre.x86_64_g64367ba.src.rpm"
2. rpm2cpio lustre-*src.rpm | cpio -idv
3. sed i 's|-with-kmp-moddir=% {kmoddir}|--with-kmp-moddir=%{kmoddir}

   --with-o2ib=/usr/src/openib|' lustre.spec

4. rpmbuild -bb --define '_sourcedir /root' lustre.spec

...
+ eval ./configure --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-tests --enable-utils --enable-modules --without-zfs --disable-dlc --with-release=2.6.32_504.8.1.el6_lustre.x86_64_g64367ba --with-linux=/lib/modules/2.6.32-504.8.1.el6_lustre.x86_64/source --with-linux-obj=/lib/modules/2.6.32-504.8.1.el6_lustre.x86_64/build --with-kmp-moddir=extra --with-o2ib=/usr/src/openib
++ ./configure --program-prefix= --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-tests --enable-utils --enable-modules --without-zfs --disable-dlc --with-release=2.6.32_504.8.1.el6_lustre.x86_64_g64367ba --with-linux=/lib/modules/2.6.32-504.8.1.el6_lustre.x86_64/source --with-linux-obj=/lib/modules/2.6.32-504.8.1.el6_lustre.x86_64/build --with-kmp-moddir=extra --with-o2ib=/usr/src/openib
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking target system type... x86_64-unknown-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking how to create a ustar tar archive... gnutar
checking whether to disable maintainer-specific portions of Makefiles... yes
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1966080
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking dependency style of gcc... gcc3
checking for rpmbuild... /usr/bin/rpmbuild
configure: Lustre base checks
==============================================================================
checking whether to configure just enough for make dist... no
checking if this distro uses dpkg... no
checking for buildid... g64367ba
checking for ranlib... (cached) ranlib
checking for ld... /usr/bin/ld -m elf_x86_64
checking for objdump... (cached) objdump
checking for strip... (cached) strip
checking size of unsigned long long... 8
checking whether to use OSD addon... no
checking whether to build Lustre docs... no
checking whether to build Lustre manpages... yes
checking for rst2man... rst2man
checking whether to build Lustre utilities... yes
checking whether to build Lustre tests... yes
checking whether to build Lustre client support... yes
checking whether mpitests can be built... no
checking whether to build Linux kernel modules... yes (linux-gnu)
checking for Linux sources... /usr/src/kernels/2.6.32-504.8.1.el6_lustre.x86_64
checking for /usr/src/kernels/2.6.32-504.8.1.el6_lustre.x86_64... yes
checking for Linux objects... /usr/src/kernels/2.6.32-504.8.1.el6_lustre.x86_64
checking for /usr/src/kernels/2.6.32-504.8.1.el6_lustre.x86_64/.config... yes
checking for /boot/kernel.h... no
checking for /var/adm/running-kernel.h... no
checking for /usr/src/kernels/2.6.32-504.8.1.el6_lustre.x86_64/include/generated/autoconf.h... no
checking for /usr/src/kernels/2.6.32-504.8.1.el6_lustre.x86_64/include/linux/autoconf.h... yes
checking for /usr/src/kernels/2.6.32-504.8.1.el6_lustre.x86_64/include/linux/version.h... yes
checking for /usr/src/kernels/2.6.32-504.8.1.el6_lustre.x86_64/include/linux/kconfig.h... yes
checking for cross compilation... no
checking for external module build target... M
checking that modules can be built at all... yes
checking for Linux kernel utsrelease... 2.6.32-504.8.1.el6_lustre.x86_64
checking for Lustre release... 2.6.32_504.8.1.el6_lustre.x86_64_g64367ba
checking for RedHat kernel release number... 66
checking for Linux kernel module package directory... extra
checking for the name of module symbol version file... Module.symvers
checking if Linux kernel was built with CONFIG_MODULES... yes
checking if Linux kernel was built with CONFIG_MODVERSIONS... yes
checking if Linux kernel was built with CONFIG_KALLSYMS... yes
checking if Linux kernel module loading is possible... yes
configure: LibCFS kernel checks
==============================================================================
checking whether to use tunable 'panic_dumplog' support... no
checking if 'struct stacktrace_ops' has 'walk_stack' field... yes
checking if 'shrinker' want self pointer in handler... yes
checking if 'ctl_table' has a 'ctl_name' field... yes
checking if Linux kernel has 'sk_sleep'... no
checking if Linux kernel has 'kstrtoul'... yes
checking if 'dump_trace' want address... no
checking if 'shrink_control' is present... no
checking if 'stacktrace_ops.warning' is exist... yes
checking for linux/uidgid.h... no
checking if 'i_uid_read' is present... no
checking if Linux kernel exports 'sock_alloc_file'... no
checking if Linux kernel was built with CONFIG_CRC32 in or as module... yes
checking if Linux kernel was built with CONFIG_CRYPTO_CRC32_PCLMUL in or as module... no
configure: WARNING:

No crc32 pclmulqdq crypto api found, enable internal pclmulqdq based crc32

checking if Linux kernel was built with CONFIG_CRYPTO_CRC32C_INTEL in or as module... yes
checking shrinker has 'count_objects'... no
checking does function hlist_add_after exist'... yes
configure: LNet kernel checks
==============================================================================
checking if 'dev_get_by_name' has two args... yes
checking whether to enable CPU affinity support... yes
checking if Linux kernel has cpu affinity support... yes
checking whether to enable tunable backoff TCP support... yes
checking if Linux kernel has tunable backoff TCP support... no
checking whether to use Compat RDMA... yes
checking whether to use any OFED backport headers... no
checking whether to enable OpenIB gen2 support... no
configure: error: can't compile with OpenIB gen2 headers under /usr/src/compat-rdma

Looks similar to https://jira.hpdd.intel.com/browse/LU-6327

Comments

[James A Simmons]

Can you post your config.log file.

[Marcin Dulak]

Attached

[James A Simmons]

Hmm. Can you try patch http://review.whamcloud.com/#/c/15386. It needs one more cleanup but it should address your issues.

[Marcin Dulak]

Running rpmbuild -bb lustre.spec against the patched lustre-2.7.55.tar.gz did not change anything for me.

[Gerrit Updater]

James Simmons ([e-mail]) uploaded a new patch: http://review.whamcloud.com/15582
Subject: LU-6790 build: handle external Intel OFED stack
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 92b7ea38e8b621dba7efb7002aa818c9d61682e3

[James A Simmons]

Marcin try this patch.

[James A Simmons]

Any news Marcin?

[Marcin Dulak]

I think this patch has no influence on rpmbuild, because autoconf is not used.
Even if I modify the configure script to include the changes from http://review.whamcloud.com/15582
and try to make sure #define HAVE_COMPAT_RDMA 1 is used, I still get the original error.
We should not be verifying such things by hand - this is a task for a CI system.

[James A Simmons]

You need to run ./configure first since it appears you can't do a rpmbuild --define '__with_o2ib=...'. Lustre doesn't build like other rpms. See LU-3953 for details about the work to make this happen.

[Marcin Dulak]

I bypass rpmbuild's defines, see step 3. on the top of this report:
sed -i 's|-with-kmp-moddir=%

{kmoddir}|--with-kmp-moddir=%{kmoddir}

--with-o2ib=/usr/src/openib|' lustre.spec
As I mentioned I've tested a ./configure patched with http://review.whamcloud.com/15582
(i.e. I modify ./configure manually and create lustre-2.7.55.tar.gz to be used by rpmbuild)
and still getting "can't compile with OpenIB gen2 headers under /usr/src/compat-rdma"
lustre-2.5.3 builds fine against /usr/src/compat-rdma and lustre's 2.5.3 kernel (https://downloads.hpdd.intel.com/public/lustre/lustre-2.5.3/el6/server/RPMS/x86_64/) this way.

[Marcin Dulak]

I'm reusing the lustre 2.7.0 server machine for other purposes.
Please setup virtualbox/vagrant and use it as a basic CI system.
The instructions are on top of the attached Vagrantfile, "vagrant up" show the configure error.
Future patches need to address the lustre source and possibly lustre.spec.

[James A Simmons]

Where can I get the Intel OFED stack from?

[Marcin Dulak]

Look into the Vagrantfile:
wget --quiet http://downloadmirror.intel.com/24625/eng/IntelIB-Basic.RHEL6-x86_64.7.3.1.0.12.tgz

[Dmitry Eremin (Inactive)]

This is because of LNET autoconf is not ready for RHEL 6.6 support. The following patch resolve this issue.

diff --git a/lnet/autoconf/lustre-lnet.m4 b/lnet/autoconf/lustre-lnet.m4
index fa5276d..e33fdda 100644
--- a/lnet/autoconf/lustre-lnet.m4
+++ b/lnet/autoconf/lustre-lnet.m4
@@ -247,7 +247,12 @@ directory which is likely in ${O2IBPATH%-*}
                                        64)
                                                EXTRA_OFED_INCLUDE="$EXTRA_OFED_INCLUDE -DCONFIG_COMPAT_RHEL_6_4" ;;
                                        65)
-                                               EXTRA_OFED_INCLUDE="$EXTRA_OFED_INCLUDE -DCONFIG_COMPAT_RHEL_6_4 -DCONFIG_COMPAT_RHEL_6_5" ;;
+                                               EXTRA_OFED_INCLUDE="$EXTRA_OFED_INCLUDE -DCONFIG_COMPAT_RHEL_6_4"
+                                               EXTRA_OFED_INCLUDE="$EXTRA_OFED_INCLUDE -DCONFIG_COMPAT_RHEL_6_5" ;;
+                                       66)
+                                               EXTRA_OFED_INCLUDE="$EXTRA_OFED_INCLUDE -DCONFIG_COMPAT_RHEL_6_4"
+                                               EXTRA_OFED_INCLUDE="$EXTRA_OFED_INCLUDE -DCONFIG_COMPAT_RHEL_6_5"
+                                               EXTRA_OFED_INCLUDE="$EXTRA_OFED_INCLUDE -DCONFIG_COMPAT_RHEL_6_6" ;;
                                esac
                        elif test "x$SUSE_KERNEL" = xyes; then
                                SP=$(grep PATCHLEVEL /etc/SuSE-release | sed -e 's/.*= *//')

[James A Simmons]

Wow. So the problem was the lack of RHEL6.6 support. In the future we need to be more careful when moving to a new distro.

[Dmitry Eremin (Inactive)]

Yes, I'd like to pay your attention on RHEL 7.x is not supported also. So, any attempt to compile Lustre with external OFED on RHEL 7.x will fail.

[Marcin Dulak]

I'm not sure who will be technically maintaining the lustre master branch repository after reading http://wiki.lustre.org/Retired_Release_Terminology , but the future CI system integrated with code reviews must build against various OFEDs, so such errors are catched by a machine.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/15582/
Subject: LU-6790 build: handle external Intel OFED stack
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 81f4797a105e2c5d6300f465d9fb9b7174d2177b

[Peter Jones]

Landed for 2.8

[James A Simmons]

Marcin the rest of the needed fix will be done under ticket LU-6912.