[LU-9612] Unable to create rpm-dkms for zfs

• Type: Bug
• Resolution: Done
• Priority: Major
• Affects Version/s: Lustre 2.10.0

Environment

OS SLES12 SP2
kernel-default-4.4.21-69.1.x86_64
nfs-kernel-server-1.3.0-26.3.x86_64
kernel-macros-4.4.21-69.1.noarch
kernel-default-devel-4.4.21-69.1.x86_64
kernel-devel-4.4.21-69.1.noarch
kernel-firmware-20160516git-17.6.noarch
kernel-source-4.4.21-69.1.noarch

Epic/Theme

zfs

• Severity: 2
• Assignee: Malcolm Cowe (Inactive)
• Reporter: Abe (Inactive)

Description

Going through the instructions on http://wiki.lustre.org/Compiling_Lustre zfs build section:

Prepare the build
Run the configure script:
cd $HOME/zfs

1. For RHEL and CentOS only, set the --spec=redhat flag.
   ./configure [--with-spec=redhat] \
   [--with-spl=<path to spl-devel> \
   [--with-linux=<path to kernel-devel>] \
   [--with-linux-obj=<path to kernel obj>]

I issued the following configure command script for suse linux os after build the spl package:

./configure --with-spec=suse --with-spl=/usr/src/spl-0.7.0 --with-linux=/usr/src/linux-4.4.21-69/kernel

It does not create zfs spec under suse folder as it only create these under redhat and generic folders:
config.status: creating rpm/Makefile
config.status: creating rpm/redhat/Makefile
config.status: creating rpm/redhat/zfs.spec
config.status: creating rpm/redhat/zfs-kmod.spec
config.status: creating rpm/redhat/zfs-dkms.spec
config.status: creating rpm/generic/Makefile
config.status: creating rpm/generic/zfs.spec
config.status: creating rpm/generic/zfs-kmod.spec
config.status: creating rpm/generic/zfs-dkms.spec
config.status: creating zfs-script-config.sh
config.status: creating zfs.release
config.status: creating zfs_config.h
config.status: executing depfiles commands
config.status: executing libtool commands

This causes the make rpm-dkms to fail as there are no spec files configured under rpm/suse folder:

make rpm-dkms errors snapshot:cp: cannot stat 'rpm/suse/zfs-dkms.spec': No such file or directory
Makefile:1149: recipe for target 'rpm-local' failed
make[2]: *** [rpm-local] Error 1
make[2]: Leaving directory '/root/zfs'
Makefile:1167: recipe for target 'srpm-common' failed
make[1]: *** [srpm-common] Error 1
make[1]: Leaving directory '/root/zfs'
Makefile:1123: recipe for target 'srpm-dkms' failed
make: *** [srpm-dkms] Error 2

This works for spl packages and not zfs

Issue Links

• is related to: LU-9675 SLES 12 SP2 server build fails when ZFS and LDISKFS OSDs are both enabled

Comments

[Malcolm Cowe (Inactive)]

There is no documentation that I am aware of to build ZFS on SLES 12 SP2 with DKMS support, and my own initial attempts have not been successful. The ZFS on Linux project's documentation doesn't have much information about SUSE.

Clients can run any supported OS, but should ideally be running the same version of Lustre. Normally, users will use a common platform across both servers and clients.

[Abe (Inactive)]

This is awesome, i was able to build the rpms for the servers and the clients:
./configure --disable-server --enable-client --disable-ldiskfs --with-linux=/usr/src/linux-4.4.21-69 --with-linux-obj=/usr/src/linux-4.4.21-69-obj/x86_64/default
At this point, have installed the rpms on the servers and applied zfs configurations without any issues and started benchmarking our platform..

One more thing, if you have any documentation on how to use dkms for installs on sles12 sp2 that would be great..
I assume that the clients needs to be running sles12 sp2 os as well correct ?

thank you so much for your support here,

Regards,
Abe

[Malcolm Cowe (Inactive)]

There's a bug somewhere in the "make rpms" process that prevents creating LDISKFS and ZFS packages at the same time. The workaround is to disable the LDISKFS support explicitly:

./configure --enable-server \
--disable-ldiskfs \
--with-linux=/usr/src/linux-4.4.21-69 \
--with-linux-obj=/usr/src/linux-4.4.21-69-obj/x86_64/default

Note the second line in the above command.

[Abe (Inactive)]

Hi Malcom,
i figured out how to add the packages on the server, basically user needs to subscribe and register the node with suse server with NFR license..
BB-02:~ # SUSEConnect -r 1FAC34FBD7DDAA
Registered SLES 12.2 x86_64
To server: https://scc.suse.com
SBB-02:~ # SUSEConnect -p
Please provide a product identifier
SBB-02:~ # SUSEConnect -p sle-sdk/12.2/x86_64
Registered sle-sdk 12.2 x86_64
To server: https://scc.suse.com

so spl and zfs components are installed ..

SBB-02:~ # rpm -qf /usr/include/libspl
libzfs2-devel-0.7.0-rc4.x86_64
SBB-02:~ # rpm -qf /usr/include/libzfs
libzfs2-devel-0.7.0-rc4.x86_64

Now we are hitting a problem with lustre compilation, these are the steps have used:
1. make distclean
2. ./configure --enable-server --with-linux=/usr/src/linux-4.4.21-69 --with-linux-obj=/usr/src/linux-4.4.21-69-obj/x86_64/default
3. make rpms

SBB-02:~/lustre-release # make rpms
make dist-gzip am__post_remove_distdir='@:'
make[1]: Entering directory '/root/lustre-release'
if test -d "lustre-2.9.58_22_gdb59ecb"; then find "lustre-2.9.58_22_gdb59ecb" -type d ! -perm -200 -exec chmod u+w {} ';' && rm -rf "lustre-2.9.58_22_gdb59ecb" ||

{ sleep 5 && rm -rf "lustre-2.9.58_22_gdb59ecb"; }

; else :; fi
test -d "lustre-2.9.58_22_gdb59ecb" || mkdir "lustre-2.9.58_22_gdb59ecb"
(cd ldiskfs && make top_distdir=../lustre-2.9.58_22_gdb59ecb distdir=../lustre-2.9.58_22_gdb59ecb/ldiskfs \
am_remove_distdir=: amskip_length_check=: am_skip_mode_fix=: distdir)

snapshot:
rm -rf $rpmbuilddir
make[1]: Entering directory '/root/lustre-release'
make[1]: Leaving directory '/root/lustre-release'
Wrote: /tmp/rpmbuild-lustre-root-DfOI5x2b/SRPMS/lustre-2.9.58_22_gdb59ecb-1.src.rpm
rpmbuilddir=`mktemp t -d rpmbuild-lustre$USER-XXXXXXXX`; \
make \
rpmbuilddir="$rpmbuilddir" rpm-local || exit 1; \
/usr/bin/rpmbuild \
--define "_tmppath $rpmbuilddir/TMP" \
--define "_topdir $rpmbuilddir" \
--define "kdir /usr/src/linux-4.4.21-69" --define "kobjdir /usr/src/linux-4.4.21-69-obj/x86_64/default" --with zfs --with lnet_dlc \
-rebuild lustre-2.9.58_22_gdb59ecb*.src.rpm || exit 1; \
cp $rpmbuilddir/RPMS//.rpm . || exit 1; \
rm -rf $rpmbuilddir
make[1]: Entering directory '/root/lustre-release'
make[1]: Leaving directory '/root/lustre-release'
Installing lustre-2.9.58_22_gdb59ecb-1.src.rpm
sh: -c: line 15: unexpected EOF while looking for matching `"'
sh: -c: line 17: syntax error: unexpected end of file
error: Summary field must be present in package: lustre-osd-zfs-kmp-default
autoMakefile:1091: recipe for target 'rpms' failed
make: *** [rpms] Error 1
SBB-02:~/lustre-release #

thanks,
Abe

[Malcolm Cowe (Inactive)]

SUSE provides the SDK add-on – I thought it was one of the standard add-ons for SLES. There might be an ISO download, but I am not sure.

I forget which packages come from the SDK versus the core OS – if you run the above command, it will tell you which packages it couldn't find, if any.

[Abe (Inactive)]

how do you enable the SLE-SDK12 add-on ?

thanks

[Malcolm Cowe (Inactive)]

The following link has the package pre-requisites for compiling ZFS and Lustre:

http://wiki.lustre.org/Compiling_Lustre#SLES_12:_Install_the_Software_Development_Tools

The following command will install the pre-reqs:

sudo zypper install asciidoc automake bc binutils-devel bison bison \
  device-mapper-devel elfutils libelf-devel flex gcc gcc-c++ git \
  glib2-tools glib2-devel hmaccalc libattr-devel libblkid-devel \
  libselinux-devel libtool libuuid-devel lsscsi make mksh ncurses-devel \
  net-tools numactl parted patchutils pciutils-devel perl pesign \
  python-devel rpm-build sysstat systemd-devel tcl tcl-devel tk tk-devel wget \
  xmlto zlib-devel libyaml-devel krb5-devel keyutils-devel net-snmp-devel

You will need to have the SLE-SDK12 add-on enabled. Additional dependencies will be resolved automatically.

[Abe (Inactive)]

cd $HOME/zfs
make pkg-utils pkg-mod ---> returns libzfs compile errors, missing ludev lib
snapshot below:

make[5]: Entering directory '/tmp/zfs-build-root-0K0ri1dq/BUILD/zfs-0.7.0/lib/libzfs'
CC libzfs_changelist.lo
CC libzfs_config.lo
CC libzfs_dataset.lo
CC libzfs_diff.lo
CC libzfs_fru.lo
CC libzfs_import.lo
CC libzfs_iter.lo
CC libzfs_mount.lo
CC libzfs_pool.lo
CC libzfs_sendrecv.lo
CC libzfs_status.lo
CC libzfs_util.lo
CCLD libzfs.la
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld: cannot find -ludev
collect2: error: ld returned 1 exit status
Makefile:631: recipe for target 'libzfs.la' failed
make[5]: *** [libzfs.la] Error 1
make[5]: Leaving directory '/tmp/zfs-build-root-0K0ri1dq/BUILD/zfs-0.7.0/lib/libzfs'
Makefile:534: recipe for target 'all-recursive' failed
make[4]: *** [all-recursive] Error 1
make[4]: Leaving directory '/tmp/zfs-build-root-0K0ri1dq/BUILD/zfs-0.7.0/lib'
Makefile:707: recipe for target 'all-recursive' failed
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory '/tmp/zfs-build-root-0K0ri1dq/BUILD/zfs-0.7.0'
Makefile:575: recipe for target 'all' failed
make[2]: *** [all] Error 2
make[2]: Leaving directory '/tmp/zfs-build-root-0K0ri1dq/BUILD/zfs-0.7.0'
error: Bad exit status from /tmp/zfs-build-root-0K0ri1dq/TMP/rpm-tmp.NljItE (%build)

RPM build errors:
Bad exit status from /tmp/zfs-build-root-0K0ri1dq/TMP/rpm-tmp.NljItE (%build)
Makefile:1183: recipe for target 'rpm-common' failed
make[1]: *** [rpm-common] Error 1
make[1]: Leaving directory '/root/zfs'
Makefile:1142: recipe for target 'rpm-utils' failed
make: *** [rpm-utils] Error 2

[Malcolm Cowe (Inactive)]

Yes, I would have expected these to have been created as part of the build process, but it depends on the options supplied to the make command for the SPL and ZFS code. 

To create the user space packages, you run this command in both of the SPL and the ZFS directories:

make pkg-utils

This is my build process for SPL and ZFS on SLES 12 SP2:

# SPL
cd $HOME/spl
sh autogen.sh # only needed the first time the code is checked out
./configure --with-linux=/usr/src/linux-4.4.59-92.17
make pkg-utils pkg-kmod
mkdir -p $HOME/releases/zfs-spl
mv $HOME/spl/*.rpm $HOME/releases/zfs-spl
cd $HOME/releases/zfs-spl
sudo rpm -ivh kmod-spl-* spl-0.7.0*.x86_64.rpm

# ZFS
cd $HOME/zfs
./configure --with-linux=/usr/src/linux-4.4.59-92.17
make pkg-utils pkg-kmod
mv $HOME/zfs/*.rpm $HOME/releases/zfs-spl
cd $HOME/releases/zfs-spl
sudo rpm -ivh kmod-zfs-[0-9].*-default-*.x86_64.rpm \
kmod-zfs-devel-[0-9].*.x86_64.rpm \
lib*.x86_64.rpm \
zfs-[0-9].*.x86_64.rpm \
zfs-dracut-[0-9].*.x86_64.rpm

 

[Abe (Inactive)]

The lib,zfs,zfs-dracut rpm are missing, the configure script should have generated these right ?

SBB-02:~/releases/zfs-spl # sudo rpm ivh kmod-spl* spl-.x86_64.rpm kmod-zfs-[0-9].default.x86_64.rpm kmod-zfs-devel-[0-9]..x86_64.rpm lib*.x86_64.rpm zfs-[0-9]..x86_64.rpm zfs-dracut-[0-9]..x86_64.rpm
error: File not found by glob: lib*.x86_64.rpm
error: File not found by glob: zfs-[0-9].*.x86_64.rpm
error: File not found by glob: zfs-dracut-[0-9].*.x86_64.rpm

[Malcolm Cowe (Inactive)]

run the following command:

rpm -qi libzfs2-devel

If the package is not installed then run, from the directory containing the ZFS RPMs you created:

sudo rpm -ivh libzfs2-devel-*.rpm

If that fails, then likely, you need to install all the library packages in the release directory. The simplest way to do that is:

sudo rpm -ivh lib*.rpm

From the output of your commands, the user-space tools might not be installed, either.  

The lustre.org documentation uses the following command to install all the pre-requisite ZFS packages:

cd $HOME/releases/zfs-spl
sudo rpm -ivh kmod-spl-* spl-*.x86_64.rpm \
kmod-zfs-[0-9].*-default-*.x86_64.rpm \
kmod-zfs-devel-[0-9].*.x86_64.rpm \
lib*.x86_64.rpm \
zfs-[0-9].*.x86_64.rpm \
zfs-dracut-[0-9].*.x86_64.rpm