Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
3
-
14250
Description
This issue was created by maloo for Andreas Dilger <andreas.dilger@intel.com>
This issue relates to the following test suite run: http://maloo.whamcloud.com/test_sets/3a80b878-eff3-11e3-a29d-52540035b04c.
It seems that the lustre-tests RPM is not being installed on the test nodes for some reason?
The sub-test lustre-initialization_1 failed with the following error in the autotest logs:
bash: line 0: cd: /usr/lib64/lustre/tests: No such file or directory
sh: mkfs.lustre: command not found
Info required for matching: lustre-initialization-1 lustre-initialization_1
Attachments
Issue Links
- is duplicated by
-
-
- Resolved
-
-
-
- Resolved
-
-
-
- Resolved
-
- is related to
-
-
- Resolved
-
-
-
- Resolved
-
- is related to
-
-
- Resolved
-
-
-
- Resolved
-
(2 is related to , 5 mentioned in)
Activity
Just to be complete about this ticket's problem, and even if we will "track follow on work to improve robustness of this area under a new ticket", my patch fixes a problem/typo causing configure to fail when [spl,zfs]-dkms packages are in "added" state, in the sense of DKMS. But the main problem is that these packages are still in tis "added" state at the time of configure step during lustre-dkms install.
This should never happen since both [spl,zfs]-dkms packages should be built/installed during their respective RPM's post-install script.
Having a look to an affected node's KickStart/install log, this could be linked to the following msgs/errors :
............................
+ yum install -y kernel-2.6.32-573.8.1.el6.x86_64 kernel-devel-2.6.32-573.8.1.el6.x86_64
Loaded plugins: fastestmirror, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:2.6.32-573.8.1.el6 will be installed
---> Package kernel-devel.x86_64 0:2.6.32-573.8.1.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
kernel x86_64 2.6.32-573.8.1.el6 updates 30 M
kernel-devel x86_64 2.6.32-573.8.1.el6 updates 10 M
Transaction Summary
================================================================================
Install 2 Package(s)
Total download size: 40 M
Installed size: 151 M
Downloading Packages:
--------------------------------------------------------------------------------
Total 4.2 MB/s | 40 MB 00:09
warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Importing GPG key 0xC105B9DE:
Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>
Package: centos-release-6-7.el6.centos.12.3.x86_64 (@anaconda-CentOS-201508042137.x86_64/6.7)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
^M Installing : kernel-devel-2.6.32-573.8.1.el6.x86_64 1/2
^M Installing : kernel-2.6.32-573.8.1.el6.x86_64 2/2
^M Verifying : kernel-2.6.32-573.8.1.el6.x86_64 1/2
^M Verifying : kernel-devel-2.6.32-573.8.1.el6.x86_64 2/2
Installed:
kernel.x86_64 0:2.6.32-573.8.1.el6 kernel-devel.x86_64 0:2.6.32-573.8.1.el6
Complete!
+ break
yuminstall zfs-dkms spl-dkms zfs
+ yuminstall zfs-dkms spl-dkms zfs
+ local 'packages=zfs-dkms spl-dkms zfs'
+ local max=5
+ local i=0
+ (( i < max ))
+ yum install -y zfs-dkms spl-dkms zfs
Loaded plugins: fastestmirror, security
Setting up Install Process
Determining fastest mirrors
* base: centos.mirror.constant.com
* extras: repos.lax.quadranet.com
* updates: mirrors.cmich.edu
Resolving Dependencies
--> Running transaction check
---> Package spl-dkms.noarch 0:0.6.5.3-1.el6 will be installed
--> Processing Dependency: dkms >= 2.2.0.2 for package: spl-dkms-0.6.5.3-1.el6.noarch
---> Package zfs.x86_64 0:0.6.5.3-1.el6 will be installed
--> Processing Dependency: spl = 0.6.5.3 for package: zfs-0.6.5.3-1.el6.x86_64
--> Processing Dependency: libzpool2 = 0.6.5.3 for package: zfs-0.6.5.3-1.el6.x86_64
--> Processing Dependency: libzfs2 = 0.6.5.3 for package: zfs-0.6.5.3-1.el6.x86_64
--> Processing Dependency: libuutil1 = 0.6.5.3 for package: zfs-0.6.5.3-1.el6.x86_64
--> Processing Dependency: libnvpair1 = 0.6.5.3 for package: zfs-0.6.5.3-1.el6.x86_64
--> Processing Dependency: libzpool.so.2()(64bit) for package: zfs-0.6.5.3-1.el6.x86_64
--> Processing Dependency: libzfs_core.so.1()(64bit) for package: zfs-0.6.5.3-1.el6.x86_64
--> Processing Dependency: libzfs.so.2()(64bit) for package: zfs-0.6.5.3-1.el6.x86_64
--> Processing Dependency: libuutil.so.1()(64bit) for package: zfs-0.6.5.3-1.el6.x86_64
--> Processing Dependency: libnvpair.so.1()(64bit) for package: zfs-0.6.5.3-1.el6.x86_64
---> Package zfs-dkms.noarch 0:0.6.5.3-1.el6 will be installed
--> Running transaction check
---> Package dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el6 will be installed
---> Package libnvpair1.x86_64 0:0.6.5.3-1.el6 will be installed
---> Package libuutil1.x86_64 0:0.6.5.3-1.el6 will be installed
---> Package libzfs2.x86_64 0:0.6.5.3-1.el6 will be installed
---> Package libzpool2.x86_64 0:0.6.5.3-1.el6 will be installed
---> Package spl.x86_64 0:0.6.5.3-1.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
spl-dkms noarch 0.6.5.3-1.el6 lustre-build 449 k
zfs x86_64 0.6.5.3-1.el6 lustre-build 323 k
zfs-dkms noarch 0.6.5.3-1.el6 lustre-build 1.9 M
Installing for dependencies:
dkms noarch 2.2.0.3-30.git.7c3e7c5.el6 addon-epel6-x86_64 77 k
libnvpair1 x86_64 0.6.5.3-1.el6 lustre-build 27 k
libuutil1 x86_64 0.6.5.3-1.el6 lustre-build 32 k
libzfs2 x86_64 0.6.5.3-1.el6 lustre-build 113 k
libzpool2 x86_64 0.6.5.3-1.el6 lustre-build 401 k
spl x86_64 0.6.5.3-1.el6 lustre-build 25 k
Transaction Summary
================================================================================
Install 9 Package(s)
Total download size: 3.3 M
Installed size: 16 M
Downloading Packages:
--------------------------------------------------------------------------------
Total 1.3 MB/s | 3.3 MB 00:02
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
^M Installing : libuutil1-0.6.5.3-1.el6.x86_64 1/9
^M Installing : libnvpair1-0.6.5.3-1.el6.x86_64 2/9
^M Installing : libzpool2-0.6.5.3-1.el6.x86_64 3/9
^M Installing : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch 4/9
^M Installing : spl-dkms-0.6.5.3-1.el6.noarch 5/9
Loading new spl-0.6.5.3 DKMS files...
It is likely that 2.6.32-573.el6.x86_64 belongs to a chroot's host
Building for 2.6.32-573.12.1.el6.x86_64 and 2.6.32-573.8.1.el6.x86_64 <<<<<<<<<<<<<<<<<<<
/usr/sbin/dkms: line 1958: /dev/fd/62: No such file or directory
/usr/sbin/dkms: line 1890: /dev/fd/62: No such file or directory
warning: %post(spl-dkms-0.6.5.3-1.el6.noarch) scriptlet failed, exit status 1 <<<<<<< spl-dkms RPM post-install script failed, leading to its DKMS build/install steps not done
Non-fatal POSTIN scriptlet failure in rpm package spl-dkms-0.6.5.3-1.el6.noarch
^M Installing : zfs-dkms-0.6.5.3-1.el6.noarch 6/9
Loading new zfs-0.6.5.3 DKMS files...
It is likely that 2.6.32-573.el6.x86_64 belongs to a chroot's host
Building for 2.6.32-573.12.1.el6.x86_64 and 2.6.32-573.8.1.el6.x86_64 <<<<<<<<<<<<<<<<<<<
/usr/sbin/dkms: line 1958: /dev/fd/62: No such file or directory
/usr/sbin/dkms: line 1890: /dev/fd/62: No such file or directory
warning: %post(zfs-dkms-0.6.5.3-1.el6.noarch) scriptlet failed, exit status 1 <<<<<<< zfs-dkms RPM post-install script failed, leading to its DKMS build/install steps not done
Non-fatal POSTIN scriptlet failure in rpm package zfs-dkms-0.6.5.3-1.el6.noarch
^M Installing : spl-0.6.5.3-1.el6.x86_64 7/9
^M Installing : libzfs2-0.6.5.3-1.el6.x86_64 8/9
^M Installing : zfs-0.6.5.3-1.el6.x86_64 9/9
^M Verifying : spl-dkms-0.6.5.3-1.el6.noarch 1/9
^M Verifying : dkms-2.2.0.3-30.git.7c3e7c5.el6.noarch 2/9
^M Verifying : libzpool2-0.6.5.3-1.el6.x86_64 3/9
^M Verifying : libuutil1-0.6.5.3-1.el6.x86_64 4/9
^M Verifying : zfs-dkms-0.6.5.3-1.el6.noarch 5/9
^M Verifying : spl-0.6.5.3-1.el6.x86_64 6/9
^M Verifying : zfs-0.6.5.3-1.el6.x86_64 7/9
^M Verifying : libzfs2-0.6.5.3-1.el6.x86_64 8/9
^M Verifying : libnvpair1-0.6.5.3-1.el6.x86_64 9/9
Installed:
spl-dkms.noarch 0:0.6.5.3-1.el6 zfs.x86_64 0:0.6.5.3-1.el6
zfs-dkms.noarch 0:0.6.5.3-1.el6
Dependency Installed:
dkms.noarch 0:2.2.0.3-30.git.7c3e7c5.el6 libnvpair1.x86_64 0:0.6.5.3-1.el6
libuutil1.x86_64 0:0.6.5.3-1.el6 libzfs2.x86_64 0:0.6.5.3-1.el6
libzpool2.x86_64 0:0.6.5.3-1.el6 spl.x86_64 0:0.6.5.3-1.el6
Complete!
+ break
yuminstall lustre-dkms lustre-osd-zfs lustre lustre-tests
+ yuminstall lustre-dkms lustre-osd-zfs lustre lustre-tests
+ local 'packages=lustre-dkms lustre-osd-zfs lustre lustre-tests'
+ local max=5
+ local i=0
+ (( i < max ))
+ yum install -y lustre-dkms lustre-osd-zfs lustre lustre-tests
Loaded plugins: fastestmirror, security
Setting up Install Process
Loading mirror speeds from cached hostfile
* base: centos.mirror.constant.com
* extras: repos.lax.quadranet.com
* updates: mirrors.cmich.edu
Resolving Dependencies
--> Running transaction check
---> Package lustre.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c will be installed
--> Processing Dependency: lustre-osd-mount for package: lustre-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c.x86_64
--> Processing Dependency: libnetsnmpmibs.so.20()(64bit) for package: lustre-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c.x86_64
--> Processing Dependency: libnetsnmphelpers.so.20()(64bit) for package: lustre-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c.x86_64
--> Processing Dependency: libnetsnmpagent.so.20()(64bit) for package: lustre-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c.x86_64
--> Processing Dependency: libnetsnmp.so.20()(64bit) for package: lustre-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c.x86_64
---> Package lustre-dkms.noarch 0:2.7.64-1.el6 will be installed
--> Processing Dependency: /usr/bin/expect for package: lustre-dkms-2.7.64-1.el6.noarch
---> Package lustre-osd-zfs.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c will be installed
---> Package lustre-tests.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c will be installed
--> Processing Dependency: lustre-iokit for package: lustre-tests-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c.x86_64
--> Running transaction check
---> Package expect.x86_64 0:5.44.1.15-5.el6_4 will be installed
---> Package lustre-iokit.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c will be installed
--> Processing Dependency: sg3_utils for package: lustre-iokit-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c.x86_64
---> Package lustre-osd-zfs-mount.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c will be installed
---> Package net-snmp-libs.x86_64 1:5.5-54.el6_7.1 will be installed
--> Running transaction check
---> Package sg3_utils.x86_64 0:1.28-8.el6 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
lustre x86_64 2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
lustre-build 569 k
lustre-dkms noarch 2.7.64-1.el6 lustre-build 12 M
lustre-osd-zfs x86_64 2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
lustre-build 92 k
lustre-tests x86_64 2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
lustre-build 8.4 M
Installing for dependencies:
expect x86_64 5.44.1.15-5.el6_4 base 256 k
lustre-iokit x86_64 2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
lustre-build 42 k
lustre-osd-zfs-mount x86_64 2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
lustre-build 7.7 k
net-snmp-libs x86_64 1:5.5-54.el6_7.1 updates 1.5 M
sg3_utils x86_64 1.28-8.el6 base 500 k
Transaction Summary
================================================================================
Install 9 Package(s)
Total download size: 24 M
Installed size: 53 M
Downloading Packages:
--------------------------------------------------------------------------------
Total 4.2 MB/s | 24 MB 00:05
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
^M Installing : lustre-osd-zfs-mount-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3 1/9
^M Installing : expect-5.44.1.15-5.el6_4.x86_64 2/9
^M Installing : lustre-dkms-2.7.64-1.el6.noarch 3/9
Loading new lustre-2.7.64 DKMS files...
/usr/sbin/dkms: line 1958: /dev/fd/62: No such file or directory <<<<<<<<<<<<<<<<<<< same error /dev/fd/62 problem than previously for [spl,zfs]-dkms
/usr/sbin/dkms: line 1890: /dev/fd/62: No such file or directory
/usr/sbin/dkms: line 1958: /dev/fd/62: No such file or directory
/usr/sbin/dkms: line 1890: /dev/fd/62: No such file or directory
configure: error: Kernel source could not be found.
It is likely that 2.6.32-573.el6.x86_64 belongs to a chroot's host
Building for 2.6.32-573.12.1.el6.x86_64 and 2.6.32-573.8.1.el6.x86_64
/usr/sbin/dkms: line 1958: /dev/fd/62: No such file or directory
/usr/sbin/dkms: line 1890: /dev/fd/62: No such file or directory
warning: %post(lustre-dkms-2.7.64-1.el6.noarch) scriptlet failed, exit status 1 <<<<<< lustre-dkms RPM post-script also fails
Non-fatal POSTIN scriptlet failure in rpm package lustre-dkms-2.7.64-1.el6.noarch
^M Installing : lustre-osd-zfs-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x 4/9
^M Installing : 1:net-snmp-libs-5.5-54.el6_7.1.x86_64 5/9
^M Installing : lustre-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g5 6/9
^M Installing : sg3_utils-1.28-8.el6.x86_64 7/9
^M Installing : lustre-iokit-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86 8/9
^M Installing : lustre-tests-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86 9/9
^M Verifying : sg3_utils-1.28-8.el6.x86_64 1/9
^M Verifying : lustre-osd-zfs-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x 2/9
^M Verifying : lustre-dkms-2.7.64-1.el6.noarch 3/9
^M Verifying : 1:net-snmp-libs-5.5-54.el6_7.1.x86_64 4/9
^M Verifying : lustre-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g5 5/9
^M Verifying : lustre-osd-zfs-mount-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3 6/9
^M Verifying : lustre-iokit-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86 7/9
^M Verifying : expect-5.44.1.15-5.el6_4.x86_64 8/9
^M Verifying : lustre-tests-2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86 9/9
Installed:
lustre.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
lustre-dkms.noarch 0:2.7.64-1.el6
lustre-osd-zfs.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
lustre-tests.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
Dependency Installed:
expect.x86_64 0:5.44.1.15-5.el6_4
lustre-iokit.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
lustre-osd-zfs-mount.x86_64 0:2.7.64-2.6.32_573.8.1.el6_lustre.gbd3d354.x86_64_g554e31c
net-snmp-libs.x86_64 1:5.5-54.el6_7.1
sg3_utils.x86_64 0:1.28-8.el6
Complete!
............................
Landed for 2.8. Let's track follow on work to improve robustness of this area under a new ticket.
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/17829/
Subject: LU-7601 build: fix typo for spl/zfs added case handler
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cb73cb5083ee2db5c6ed607c44e1002788b4eee6
Faccini Bruno (bruno.faccini@intel.com) uploaded a new patch: http://review.whamcloud.com/17829
Subject: LU-7601 build: fix typo for spl/zfs added case handler
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 554e31c64a2f8762598cfc0865772b818edcdce9
After having hands-on an isolated node that suffered the same failure, I have found that the "configure" failure comes from the auto-generated "lustre-dkms_post-add.sh" script (from LU-1032 latest patch to handle Client case) causing wrong configure parameters construction, when zfs/spl DKMS RPMs have only been added ...
Will push a patch to fix this.
And a workaround would be to build/install ZFS/SPL DKMS RPMs before to install Lustre DKMS RPM.
It is true that any failure/lack causing lustre-dkms RPM configure step to early exit is likely to cause such later consequence.
As part of a strengthening effort, I can also add the necessary error handling stuff to catch configure errors in the lustre-dkms RPM scripts.
Minh, any chance to help Dmitry debug this failure?
Dmitry, is it possible to make a patch to the autotest or DKMS script that will print out the installed modules and/or other information needed to debug this failure?
I think this is TEI issue. Originally I though that this happens because of using autoconf variable without proper initialization. But looking into this more deeply I think in this case we have an issue with unresolved dependency. When DKMS package is installing it launch the configure script for autoconf variables initialization. If this script fails because of any reason the dkms.conf became broken and we get mentored above error messages. So, I'd like to understand the reason of configure fail to resolve this ticket.
Can you help me to find the machine which have this issue and not changed (re-installed) yet to login and manually understand why the configure script fails?
I assume that just few packages missed or not installed yet.
Here is the hint in the console
Enabling /etc/fstab swaps: Adding 16465916k swap on /dev/sda2. Priority:-1 ext
ents:1 across:16465916k
[ OK ]^M^M
Entering non-interactive startup^M
Calling the system activity data collector (sadc)... ^M
FATAL: Module zfs not found.^M
with^M
'/kernel', '/updates', or '/extra' in record #0.^M
dkms.conf: Error! Directive 'DEST_MODULE_LOCATION' does not begin with^M
'/kernel', '/updates', or '/extra' in record #1.^M
dkms.conf: Error! Directive 'DEST_MODULE_LOCATION' does not begin with^M
'/kernel', '/updates', or '/extra' in record #2.^M
dkms.conf: Error! Directive 'DEST_MODULE_LOCATION' does not begin with^M
'/kernel', '/updates', or '/extra' in record #3.^M
dkms.conf: Error! Directive 'DEST_MODULE_LOCATION' does not begin with^M
'/kernel', '/updates', or '/extra' in record #4.^M
dkms.conf: Error! Directive 'DEST_MODULE_LOCATION' does not begin with^M
'/kernel', '/updates', or '/extra' in record #5.^M
dkms.conf: Error! Directive 'DEST_MODULE_LOCATION' does not begin with^M
'/kernel', '/updates', or '/extra' in record #6.^M
dkms.conf: Error! Directive 'DEST_MODULE_LOCATION' does not begin with^M
'/kernel', '/updates', or '/extra' in record #7.^M
dkms.conf: Error! Directive 'DEST_MODULE_LOCATION' does not begin with^M
'/kernel', '/updates', or '/extra' in record #8.^M
please make sure zfs built correctly, ie run zfs command.
Follow-on for this ticket, to add/enhance error handling/reporting (mainly during configure step) during processing of lustre[-client]-dkms package content by DKMS framework, is being tracked as part of
LU-7679.