[LU-9801] kmod-lustre-osd-zfs install failure due to missing packages Created: 27/Jul/17 Updated: 08/Aug/17 Resolved: 08/Aug/17 |
|
| Status: | Resolved |
| Project: | Lustre |
| Component/s: | None |
| Affects Version/s: | Lustre 2.10.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major |
| Reporter: | Abe | Assignee: | Malcolm Cowe (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Severity: | 3 |
| Rank (Obsolete): | 9223372036854775807 |
| Description |
|
Hi, we are unable to install this kmod rpm on centos 7 or redhat 7 os due to missing packages: os: yum install kmod-lustre-osd-zfs-2.10.0-1.el7.x86_64.rpm
|
| Comments |
| Comment by Abe [ 27/Jul/17 ] |
|
os: centos uname -r kernel installed: |
| Comment by Malcolm Cowe (Inactive) [ 27/Jul/17 ] |
|
Is the kmod-lustre-osd-zfs-2.10.0-1.el7.x86_64.rpm package taken from the Lustre download site? And is the kernel that is installed taken from the regular OS distribution? If so, then I think there is a compatibility issue between the kmod-osd-zfs binary from the Lustre download site and the kernel that's been installed. This is most likely because the kmod-osd-zfs binary packages is compiled against a Lustre-patched kernel, not a stock kernel. Also, is ZFS installed? And if so, how did you install it? It might be that the kmod package cannot resolve the symbols for the ZFS binaries – again, probably because of differences in the kernels that were used to compile the modules. Could you try the following process as an alternative installation strategy? It's lifted from a different document I'm preparing, os might need a couple of tweaks, but should be more or less sound. The idea is to install the required kernel, install SPL and ZFS using DKMS, then install the Lustre modules with DKMS and the remaing Lustre packages. It may also be possible to use the ZFS binaries from the ZFS on Linux project, but I haven't tried that yet. Outline instructions (may need to be adjusted)Install the kernel packages that match the latest supported version for the Lustre release: yum install \ kernel \ kernel-devel \ kernel-headers \ kernel-tools \ kernel-tools-libs \ kernel-tools-libs-devel It may be necessary to specify the kernel package version number in order to ensure that a kernel that is compatible with Lustre is installed. For example, Lustre 2.10.0 has support for RHEL kernel 3.10.0-514.21.1.el7: yum install \ kernel-3.10.0-514.21.1.el7 \ kernel-devel-3.10.0-514.21.1.el7 \ kernel-headers-3.10.0-514.21.1.el7 \ kernel-tools-3.10.0-514.21.1.el7 \ kernel-tools-libs-3.10.0-514.21.1.el7 \ kernel-tools-libs-devel-3.10.0-514.21.1.el7 Refer to the Lustre Changelog for the list of supported kernels. Ensure that a persistent hostid has been generated on the machine. If necessary, generate a persistent hostid (needed to help protect zpools against simultaneous imports on multiple servers). For example: hid=`[ -f /etc/hostid ] && od -An -tx /etc/hostid|sed 's/ //g'` [ "$hid" = `hostid` ] || genhostid Reboot the node. reboot Install EPEL (this might be optional but I have not checked): yum -y install epel-release Follow the instructions from the ZFS on Linux project to install the ZFS YUM repository definition. Use the DKMS package repository (the default). For example: yum -y install \
http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm
Clean the YUM cache: yum clean all Install ZFS: yum install spl-dkms zfs-dkms Download and install the following packages from https://downloads.hpdd.intel.com/public/lustre/lustre-2.10.0/el7/server/RPMS/x86_64/: lustre-dkms-2.10.0-1.el7.noarch.rpm lustre-osd-zfs-mount-2.10.0-1.el7.x86_64.rpm lustre-2.10.0-1.el7.x86_64.rpm lustre-resource-agents-2.10.0-1.el7.x86_64.rpm |
| Comment by Abe [ 27/Jul/17 ] |
|
we are still missing these packages during the download: Also, where can retrieve the spl-dkms and zfs-dkms installed kernel: thanks. |
| Comment by Malcolm Cowe (Inactive) [ 28/Jul/17 ] |
|
Please provide the complete list of command that you are using to install the servers, and the complete list of Lustre packages that you are trying to install. My instructions assume that you are not using LDISKFS, and are using the OS kernels, not the Lustre patched kernels. If you require everything (including LDISKFS), then the instructions are slightly different. The spl-dkms and zfs-dkms packages are taken from the ZFS on Linux project: The instructions for installing ZFS on RHEL / CentOS are here: https://github.com/zfsonlinux/zfs/wiki/RHEL-%26-CentOS If you follow my instructions, it will set up the servers such that the ZFS and SPL packages are pulled in via YUM automatically. i.e., running the following will set up a YUM repo description on the server that points to the ZFS on Linux repo: yum -y install \ yum clean all yum install spl-dkms zfs-dkms
|
| Comment by Abe [ 28/Jul/17 ] |
|
Hi Malcom, sudo yum install http://download.zfsonlinux.org/epel/zfs-release.el7_3.noarch.rpm Repository zfs is listed more than once in the configuration
thanks, |
| Comment by Abe [ 28/Jul/17 ] |
|
Also, the exact command ran on the system: yum install kmod-lustre-2.10.0-1.el7.x86_64.rpm kmod-lustre-tests-2.10.0-1.el7.x86_64.rpm kmod-lustre-osd-zfs-2.10.0-1.el7.x86_64.rpm |
| Comment by Malcolm Cowe (Inactive) [ 30/Jul/17 ] |
|
There are two items of note in the output you supplied:
The fact that there are more than one repo definitions indicates that there is a conflict when trying to identify the packages to download. It might be something simple like stale metadata, in which case this can be cleaned using the following command: yum clean all If this does not resolve the issue, then the conflict exists within the configuration of YUM. Run the following command to get details of the repos: yum repoinfo zfs This will return a high level summary of the ZFS repo. If there is more than one definition, I'm not sure if it will return both. My lab system returns the following output as an example: [root@ctb-el73 ~]# yum repoinfo zfs Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: centos.mirror.ausnetservers.net.au * epel: mirror.as24220.net * extras: mirror.ventraip.net.au * updates: mirror.intergrid.com.au Repo-id : zfs/x86_64 Repo-name : ZFS on Linux for EL7 - dkms Repo-status : enabled Repo-revision: 1501111462 Repo-updated : Wed Jul 26 19:24:24 2017 Repo-pkgs : 67 Repo-size : 39 M Repo-baseurl : http://download.zfsonlinux.org/epel/7.3/x86_64/ Repo-expire : 604,800 second(s) (last: Fri Jul 28 23:27:03 2017) Filter : read-only:present Repo-filename: /etc/yum.repos.d/zfs.repo repolist: 67 You need to look in the /etc/yum.repos.d directory for files that contain zfs repo definitions, and disable / remove the one that conflicts with the ZFS on Linux zfs-release rpm. Also, check that the zfs-release package has not been compromised, using rpm verify ( rpm -Vv). For example: [root@ctb-el73 ~]# rpm -Vv zfs-release ......... /etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux ......... c /etc/yum.repos.d/zfs.repo The output should be similar to the above. The first field in the output should all dots. If not, then the file has been altered, overwritten or removed. You can also check the contents of the ZFS repository using the reposync command: reposync -r zfs -un On a correctly configured system, the output will be similar to the following (version numbers may differ slightly if there have been any updates): [root@ctb-el73 ~]# reposync -r zfs -un http://download.zfsonlinux.org/epel/7.3/x86_64/libnvpair1-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/libuutil1-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/libzfs2-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/libzfs2-devel-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/libzpool2-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/spl-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/debug/spl-debuginfo-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/spl-dkms-0.7.0-1.el7_3.noarch.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/zfs-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/debug/zfs-debuginfo-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/zfs-dkms-0.7.0-1.el7_3.noarch.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/zfs-dracut-0.7.0-1.el7_3.x86_64.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/zfs-release-1-4.el7_3.centos.noarch.rpm http://download.zfsonlinux.org/epel/7.3/x86_64/zfs-test-0.7.0-1.el7_3.x86_64.rpm Note that both spl-dkms and zfs-dkms are listed in this repository.
|
| Comment by Malcolm Cowe (Inactive) [ 31/Jul/17 ] |
|
Complete and comprehensive instructions for building Lustre are available on the lustre.org wiki: http://wiki.lustre.org/Compiling_Lustre This document provides a description for creating Lustre packages for servers and clients across several operating systems, and includes ZFS, LDISKFS, DKMS build options and descriptions for 3rd party network drivers such as Mellanox OFED (MOFED) and Intel Omni-path Architecture (OPA). |
| Comment by Abe [ 31/Jul/17 ] |
|
Hi Malcom, It look like there was one repo for ZFS which previously add with Intel IML release, i have removed that
repolist: 0 after zfs / iml removal: oot@OSS1-A212 x86_64]# yum repoinfo zfs
[root@OSS1-A212 x86_64]# reposync -r zfs -un snapshot of the kmod install: lustre-2.10/downloads.hpdd.intel.com/public/lustre/latest-feature-release/el7/server/RPMS/x86_64[?1034h[root@OSS1-A212 x86_64]# yum install kmod*rpm
thanks, |
| Comment by Abe [ 31/Jul/17 ] |
|
Also, the spl and zfs kmod rpm are installed: |
| Comment by Malcolm Cowe (Inactive) [ 01/Aug/17 ] |
|
The kmod-lustre-osd-zfs RPM distributed on the Lustre download site doesn't work with the ZoL packages. The kmod-lustre-osd-zfs RPM is compiled against ZFS 0.6.5.x, whereas the latest release version is 0.7.0. While the ZFS version could be downgraded to match Lustre 2.10.0 binaries, this is not sufficient to resolve the ksym dependency problem. The kmod-lustre-osd-zfs package is also compiled against the Lustre-patched kernel, and uses ZFS RPMs that are also created on the build environment. This means that the ksym dependencies don't match up with the ZoL distribution kmod RPMs. Due to the licensing incompatibility between CDDL (ZFS) and GPL (kernel and Lustre), the binary kmod ZFS packages used to help compile Lustre are not distributed. The Lustre-patched kernel breaks KABI compatibility with kernel modules compiled against the standard RHEL kernel for the same OS release. This means that the ZoL kmod binary RPMS won't load on a lustre-patched kernel, and that the Lustre kmod binary RPMs compiled against a Lustre-patched kernel won't load on a standard RHEL kernel. The unfortunate side-effect of all this is that the kmod-lustre-osd-zfs binary RPM in the Lustre distribution is not usable by most users, which means that you need to use the DKMS packages for Lustre, SPL and ZFS, per the earlier comment, or compile all of the packages from source. To use the DKMS packages, you will need to edit /etc/yum.repos.d/zfs.repo, disable the zfs-kmod repo and enable the zfs repo. You may also need to remove the existing ZFS packages.
|
| Comment by Abe [ 01/Aug/17 ] |
|
Hi Malcom, rpm -e kmod-zfs-0.7.0-1.el7_3.x86_64 Tried also to remove these with dkms: Any suggestions ? thanks, |
| Comment by Abe [ 01/Aug/17 ] |
|
now zfs 0.7 version has been removed: now, tried to add lustre 2.10.0 via dkms and it is not |
| Comment by Abe [ 01/Aug/17 ] |
|
yum repo has been updated to use dkms as well |
| Comment by Abe [ 02/Aug/17 ] |
|
I have installed lustre/zfs/spl using the following steps: 4. hid=`[ -f /etc/hostid ] && od -An -tx /etc/hostid|sed 's/ //g'` However, not able to run the zfs commands like zpool: thanks, |
| Comment by Malcolm Cowe (Inactive) [ 02/Aug/17 ] |
|
One of the side-effects of using DKMS is that it only builds the kernel modules – all of the user-space tools need to be installed as well. I thought that when the spl-dkms and zfs-dkms RPMs were installed, YUM would have also installed the user-space packages as dependencies but that does not seem to have happened. The following command should rectify the problem: yum -y install zfs In fact, I double-checked on my lab system, and it looks like all of the required packages are dependencies of zfs, so the above command should be all that is ever required, even on a clean system. On step 8, the --skip-broken flag should not be required. This implies that some of the required packages are not being installed. Can you share what happens when the --skip-broken flag is not used?
|
| Comment by Abe [ 02/Aug/17 ] |
|
when --skip-broken flag is not set, install fails due to kmod-lustre-tests rpm missing, OSS1-A212 x86_64]# yum install lustre*rpm
[root@OSS1-A212 x86_64]# yum install kmod-lustre-tests-2.10.0-1.el7.x86_64.rpm
....
|
| Comment by Abe [ 02/Aug/17 ] |
|
Hi Malcolm, thanks, |
| Comment by Malcolm Cowe (Inactive) [ 03/Aug/17 ] |
|
Don't try to install all of the RPMs in the Lustre repository – you end up installing more than you are likely to need. Also, if you don't resolve the dependencies, then some of the packages will fail to work – the Lustre tests have applications that use MPI, for example. I'd suggest not installing the lustre test packages unless specifically required for the server, and if they are required, then also install the dependencies. When using the DKMS packages, you have to use the DKMS kernel modules for Lustre as well. There's a known issue with the kmod-lustre-sd-zfs RPM in the download directory and it will not work properly with ZFS on Linux RPMs. Instead, I would recommend installing the following Lustre server packages (after installing ZFS DKMS): lustre-dkms-2.10.0-1.el7.noarch.rpm lustre-osd-zfs-mount-2.10.0-1.el7.x86_64.rpm lustre-2.10.0-1.el7.x86_64.rpm lustre-resource-agents-2.10.0-1.el7.x86_64.rpm The lustre-dkms package will compile and install the kernel modules, and the other packages will install the Lustre user-space utilities. |
| Comment by Abe [ 03/Aug/17 ] |
|
Hi Malcolm, the suggested lustre rpm install worked fine, however not able to run zfs commands .. Options: thanks, |
| Comment by Abe [ 04/Aug/17 ] |
|
Hi Malcolm, thanks, |
| Comment by Abe [ 07/Aug/17 ] |
|
Hi Malcolm, dkms status Options: For more information see mkfs(8). thanks, |
| Comment by Malcolm Cowe (Inactive) [ 07/Aug/17 ] |
|
DKMS does not install user-space commands, it only compiles and installs the kernel modules. This is not a Lustre issue. You just need to install the additional packages that have the command-line utilities. Use the following command to install ZFS: yum install zfs This will resolve all of the dependencies, including the user-space packages that contain the zfs and zpool command lines. Then use something like the following command to install Lustre: yum localinstall \ lustre-dkms-2.10.0-1.el7.noarch.rpm \ lustre-osd-zfs-mount-2.10.0-1.el7.x86_64.rpm \ lustre-2.10.0-1.el7.x86_64.rpm \ lustre-resource-agents-2.10.0-1.el7.x86_64.rpm This will install the kernel modules using DKMS, plus the Lustre user-space utilities and ZFS OSD mount support. The above command assumes that the packages have been downloaded to a local directory. Otherwise amend the command to use install instead of localinstall. I strongly recommend using the latest ZFS version (0.7.0) in order to get the benefit of the metadata performance improvements.
|
| Comment by Abe [ 08/Aug/17 ] |
|
it works now, have the 0.7 zfs/spl on another node. dkms status I think we can close this ticket now.. @mgs-200 ~]# zpool create [-fnd] [-o property=value] ... thank you Malcolm, |
| Comment by Malcolm Cowe (Inactive) [ 08/Aug/17 ] |
|
Hi Abe, Thanks for the update. Glad to hear you've achieved a successful outcome. |
| Comment by Malcolm Cowe (Inactive) [ 08/Aug/17 ] |
|
Customer success in installing Lustre + ZFS. Closing per customer recommendation. |