Details
-
Bug
-
Resolution: Fixed
-
Blocker
-
None
-
None
-
3
-
9223372036854775807
Description
RHEL 9.1 zfs test session failed:
https://testing.whamcloud.com/test_sessions/1dd26fcc-a33f-498f-a2a9-8eb743911601
2023-02-14T06:43:02 onyx-70vm4 - zfs successfully installed 2023-02-14T06:43:02 onyx-70vm4 - Yum installing kmod-lustre-osd-zfs... 2023-02-14T06:43:02 onyx-70vm4 - Attempt 1 2023-02-14T06:43:04 onyx-70vm3 - yum install completed: out=[Updating Subscription Management repositories. Last metadata expiration check: 0:12:27 ago on Tue 14 Feb 2023 06:30:36 AM UTC. (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) ] err=[Error: Problem: package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(__cv_broadcast) = 0xc93ea281, but none of the providers can be installed - package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(arc_add_prune_callback) = 0x1363912f, but none of the providers can be installed - package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(arc_buf_size) = 0x4f87e6b0, but none of the providers can be installed - package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(arc_remove_prune_callback) = 0x1ab2d851, but none of the providers can be installed --------8<--------
Attachments
Issue Links
Activity
"Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50746/
Subject: LU-16657 build: don't need to use zfs kmodtool on RHEL9.1
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8a967f69e1f7f3e5b72766ae47db3bf64c76bac0
"Minh Diep <mdiep@whamcloud.com>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50746
Subject: LU-16657 build: don't need to use zfs kmodtool on RHEL9.1
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 21bb46e6b7bb8d392fda8cec5aaa25304eed18f8
This does not look like an LJB issue. LJB is installing the kernel from the build, setting the proper default kernel and rebooting the VM to apply the kernel change (Jian's build from above no longer exists so I used https://build.whamcloud.com/view/Reviews/job/lustre-reviews/94201/)
[root@onyx-22vm1 ~]# uname -a Linux onyx-22vm1.onyx.whamcloud.com 5.14.0-162.18.1_lustre.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 11 20:39:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
LJB installed the expected repo
[root@onyx-22vm1 ~]# cat /etc/yum.repos.d/lustre-build.repo [lustre-build] name=lustre-build baseurl=http://mgmt/cobbler/localmirror/hudson/lustre-reviews/94201/arch-x86_64_build_type-server_distro-el9.1_ib_stack-inkernel enabled=1 priority=1 gpgcheck=0
When I install zfs, yum installs zfs-dkms as Jian saw. If I install kmod-zfs I get lots of 'nothing provides ksym' errors as Jian saw also.
... - nothing provides ksym(__SCT__cond_resched) = 0xc07351b3 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__SCT__might_resched) = 0xe2c17b5d needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__SCT__preempt_schedule) = 0x48d88a2c needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 ...
I looked at the console log for the build of the el9.1 server and see a lot of messages like this. Perhaps this has something to do with it?
11:47:46 ******************************************************************************** 11:47:46 *********************** KERNEL ABI COMPATIBILITY WARNING *********************** 11:47:46 ******************************************************************************** 11:47:46 The following kernel symbols are not guaranteed to remain compatible with 11:47:46 future kernel updates to this RHEL release: 11:47:46 11:47:46 __SCT__cond_resched 11:47:46 __SCT__preempt_schedule_notrace 11:47:46 ___ratelimit 11:47:46 __bdev_dax_supported
The issue was narrowed down to Jenkins build system and/or ljb tool.
Hi mdiep and mkvardakov, could you please investigate further? Thank you.
In order to debug lbuild build_spl_zfs(), I performed the commands step by step as follows on my local RHEL 9.1 vm node but found kmod-zfs rpm could be successfully built and installed:
# rpm -qa | grep zfs # mkdir /root/BUILD # cd BUILD/ # mkdir -p RPMS RPMS/x86_64 RPMS/noarch BUILD SOURCES SPECS SRPMS # cd # git clone https://github.com/zfsonlinux/zfs.git # cd zfs # git checkout -b lbuild zfs-2.1.5 # bash autogen.sh # ./configure --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64 --with-linux-obj=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64 # make dist # cd ../ # ln -f zfs/zfs-2.1.5.tar.gz /root/BUILD/SOURCES # ln -f zfs/scripts/kmodtool /root/BUILD/SOURCES # rpmbuild -ba zfs/rpm/generic/zfs.spec --nodeps -v --define '_use_internal_dependency_generator 0' --define 'require_kdir /root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64' --define 'kver 5.14.0-162.22.2_lustre.el9.x86_64' --define 'kernels 5.14.0-162.22.2_lustre.el9.x86_64' --define '_tmppath /var/tmp' --define 'kernelbuildroot /' --define '_topdir /root/BUILD' # rpmbuild -ba zfs/rpm/generic/zfs-kmod.spec --nodeps -v --define '_use_internal_dependency_generator 0' --define 'require_kdir /root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64' --define 'kver 5.14.0-162.22.2_lustre.el9.x86_64' --define 'kernels 5.14.0-162.22.2_lustre.el9.x86_64' --define '_tmppath /var/tmp' --define 'kernelbuildroot /' --define '_topdir /root/BUILD' # rpmbuild -ba zfs/rpm/generic/zfs-dkms.spec --nodeps -v --define '_use_internal_dependency_generator 0' --define 'require_kdir /root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64' --define 'kver 5.14.0-162.22.2_lustre.el9.x86_64' --define 'kernels 5.14.0-162.22.2_lustre.el9.x86_64' --define '_tmppath /var/tmp' --define 'kernelbuildroot /' --define '_topdir /root/BUILD' # cd /root/BUILD/RPMS/x86_64/ # rpm -ivh zfs-2.1.5-1.el9.x86_64.rpm kmod-zfs-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm kmod-zfs-devel-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm libzfs5-2.1.5-1.el9.x86_64.rpm libzfs5-devel-2.1.5-1.el9.x86_64.rpm libzpool5-2.1.5-1.el9.x86_64.rpm libnvpair3-2.1.5-1.el9.x86_64.rpm libuutil3-2.1.5-1.el9.x86_64.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:libnvpair3-2.1.5-1.el9 ################################# [ 13%] 2:libuutil3-2.1.5-1.el9 ################################# [ 25%] 3:libzfs5-2.1.5-1.el9 ################################# [ 38%] 4:libzpool5-2.1.5-1.el9 ################################# [ 50%] 5:kmod-zfs-5.14.0-162.22.2_lustre.e################################# [ 63%] 6:zfs-2.1.5-1.el9 ################################# [ 75%] Created symlink /etc/systemd/system/zfs-import.target.wants/zfs-import-cache.service → /usr/lib/systemd/system/zfs-import-cache.service. Created symlink /etc/systemd/system/zfs.target.wants/zfs-mount.service → /usr/lib/systemd/system/zfs-mount.service. Created symlink /etc/systemd/system/zfs.target.wants/zfs-share.service → /usr/lib/systemd/system/zfs-share.service. Created symlink /etc/systemd/system/zed.service → /usr/lib/systemd/system/zfs-zed.service. Created symlink /etc/systemd/system/zfs.target.wants/zfs-zed.service → /usr/lib/systemd/system/zfs-zed.service. Created symlink /etc/systemd/system/multi-user.target.wants/zfs.target → /usr/lib/systemd/system/zfs.target. Created symlink /etc/systemd/system/zfs.target.wants/zfs-import.target → /usr/lib/systemd/system/zfs-import.target. Created symlink /etc/systemd/system/zfs-volumes.target.wants/zfs-volume-wait.service → /usr/lib/systemd/system/zfs-volume-wait.service. 7:libzfs5-devel-2.1.5-1.el9 ################################# [ 88%] 8:kmod-zfs-devel-5.14.0-162.22.2_lu################################# [100%] # rpm -qa | grep kmod-zfs kmod-zfs-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64 kmod-zfs-devel-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64
The steps for manually building and installation of zfs are straightforward:
# git clone https://github.com/zfsonlinux/zfs.git # cd zfs # git co zfs-2.1.5 # git switch -c zfs-2.1.5 # sh autogen.sh # ./configure --with-spec=redhat --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.12.1.el9_1/linux-5.14.0-162.12.1_lustre.el9.x86_64 # make rpms # rpm -ivh zfs-2.1.5-1.el9.x86_64.rpm kmod-zfs-2.1.5-1.el9.x86_64.rpm kmod-zfs-devel-2.1.5-1.el9.x86_64.rpm libzfs5-2.1.5-1.el9.x86_64.rpm libzfs5-devel-2.1.5-1.el9.x86_64.rpm libzpool5-2.1.5-1.el9.x86_64.rpm libnvpair3-2.1.5-1.el9.x86_64.rpm libuutil3-2.1.5-1.el9.x86_64.rpm
However, in lbuild:build_spl_zfs(), the steps are more complicated:
+ git clone -n /data/gerrit_mirror/fs/zfs.git zfs + git checkout -b lbuild zfs-2.1.5 + bash autogen.sh + ./configure --with-linux=/var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/reused/usr/src/kernels/5.14.0-162.22.2_lustre.el9.x86_64 --with-linux-obj=/var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/reused/usr/src/kernels/5.14.0-162.22.2_lustre.el9.x86_64 + make dist + ln -f zfs/zfs-2.1.5.tar.gz /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/SOURCES + ln -f zfs/scripts/kmodtool /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/SOURCES/ + rpmbuild -ba zfs/rpm/generic/zfs.spec --nodeps ... + rpmbuild -ba zfs/rpm/generic/zfs-kmod.spec --nodeps ... + rpmbuild -ba zfs/rpm/generic/zfs-dkms.spec --nodeps ...... + rpm2cpio /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/RPMS/x86_64/kmod-zfs-devel-2.1.5-1.el9.x86_64.rpm + cpio -id + CONFIGURE_FLAGS='--with-zfs-obj=...... --with-zfs=...... --with-spl-obj=...... --with-spl=...... --with-zfs-devel=...... ' + OSDZFSRPM=true + return 0
Since Jenkins build system uses lbuild to build packages, we have to investigate the lbuild codes, which needs more time.
Hi mdiep, do you have any advice?
Hi adilger,
I found even installing kmod-zfs first, it also failed on the el9.1 node provisioned by "ljb":
[root@trevis-86vm6 ~]# yum install -y kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 Updating Subscription Management repositories. Last metadata expiration check: 3:14:21 ago on Mon 03 Apr 2023 06:13:33 PM UTC. Error: Problem: conflicting requests - nothing provides ksym(__SCT__cond_resched) = 0xc07351b3 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__SCT__might_resched) = 0xe2c17b5d needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__SCT__preempt_schedule) = 0x48d88a2c needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__alloc_pages) = 0x26458776 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__alloc_percpu) = 0x949f7342 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__bitmap_weight) = 0x63c4d61f needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 <~snip~>
I didn't hit the above issue on my local VirtualBox RHEL 9.1 vm node. Manually building, installing (zfs and Lustre) and running basic sanity tests with FSTYPE=zfs passed.
Landed for 2.16