[LU-16657] installing kmod-lustre-osd-zfs failed for RHEL 9.1 server

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.16.0
• Type: Bug
• Priority: Blocker
• Reporter: Jian Yu
• Assignee: Minh Diep
• Resolution: Fixed
• Severity: 3

Description

RHEL 9.1 zfs test session failed:
https://testing.whamcloud.com/test_sessions/1dd26fcc-a33f-498f-a2a9-8eb743911601

2023-02-14T06:43:02 onyx-70vm4 - zfs successfully installed
2023-02-14T06:43:02 onyx-70vm4 - Yum installing kmod-lustre-osd-zfs...
2023-02-14T06:43:02 onyx-70vm4 - Attempt 1
2023-02-14T06:43:04 onyx-70vm3 - yum install completed: out=[Updating Subscription Management repositories.
Last metadata expiration check: 0:12:27 ago on Tue 14 Feb 2023 06:30:36 AM UTC.
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
] err=[Error: 
 Problem: package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(__cv_broadcast) = 0xc93ea281, but none of the providers can be installed
  - package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(arc_add_prune_callback) = 0x1363912f, but none of the providers can be installed
  - package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(arc_buf_size) = 0x4f87e6b0, but none of the providers can be installed
  - package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(arc_remove_prune_callback) = 0x1ab2d851, but none of the providers can be installed
--------8<--------

Issue Links

• is related to: LU-16006 can't install kmod-lustre-osd-zfs
• is related to: LU-16626 Don't use or check for python2 for lu...

Comments

[Michael Kvardakov]

yujian I could not reproduce the issue. Could restart the build, please?

[Jian Yu]

Sure, Michael. I just retriggered the failed test session in https://review.whamcloud.com/49921.

[Peter Jones]

mkvardakov did the re-run test get you what you need to know?

[Michael Kvardakov]

pjones not much. The issue is duplicate of LU-16006 which is fixed, I guess. Jian rebased the code but that did not help.

[Minh Diep]

has this ever worked before? this is new rhel9.1 server support

[Minh Diep]

I took a quick look at the build log. I don't think this is DCO issue. it will require a large amount of time for me to dig into this

[Minh Diep]

I update kernel-devel, kernel... on the builder el9.1 but still not helping. Unless this works in manual build, I'd say this is not DCO issue. yujian please open LU ticket so cfaber can assign someone to look into this further.

[Jian Yu]

Thank you, Minh. Let me try to build and install manually.

[Jian Yu]

Hi mdiep,
After building zfs, I got the following rpms:

# ls *.x86_64.rpm | grep -v debug
kmod-zfs-2.1.9-1.el9.x86_64.rpm
kmod-zfs-devel-2.1.9-1.el9.x86_64.rpm
libnvpair3-2.1.9-1.el9.x86_64.rpm
libuutil3-2.1.9-1.el9.x86_64.rpm
libzfs5-2.1.9-1.el9.x86_64.rpm
libzfs5-devel-2.1.9-1.el9.x86_64.rpm
libzpool5-2.1.9-1.el9.x86_64.rpm
zfs-2.1.9-1.el9.x86_64.rpm
zfs-test-2.1.9-1.el9.x86_64.rpm

And after installing them, I can successfully install kmod-lustre-osd-zfs. Could you please check if any zfs related rpms were not installed while provisioning the node?

[Minh Diep]

Hi yujian 

Could you tell me the steps you did? I'd like to compare with lbuild. there must be something new that we missed

[Jian Yu]

After building and installing the patched kernel, I built zfs as follows and installed the rpms listed in the previous comment:

# sh autogen.sh
# ./configure --with-spec=redhat --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.12.1.el9_1/linux-5.14.0-162.12.1_lustre.el9.x86_64
# make rpms

Then I built Lustre as follows and installed the rpms successfully:

# sh autogen.sh
# ./configure --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.12.1.el9_1/linux-5.14.0-162.12.1_lustre.el9.x86_64 --with-zfs=/root/zfs --without-ldiskfs
# make rpms

[Minh Diep]

but this is not the same kernel as in the patch. could you try on 

5.14.0-162.18.1_lustre.el9.x86_64

[Jian Yu]

I already have a built kernel 5.14.0-162.12.1 on my local RHEL 9.1 vm node, so I didn't take another 4 hours to build kernel 5.14.0-162.18.1.
IMHO, there will be no problem for it.

[Jian Yu]

Hi mdiep,
I ran the following command to provision an el9.1 server node:

ljb install -d el9.1 -a x86_64 -p test -j lustre-reviews -b 93153 -t server -i inkernel -v -n trevis-86vm6

As per the node-provisioning log on Maloo, I ran "yum install -y zfs" on the node. Then I found kmod-zfs rpm was not installed, which is the root cause of this ticket.

[Andreas Dilger]

Jian, can you please push a patch that adds "Requires: kmod-zfs" (or possibly "Requires: kmp-zfs" for SLES) to the kmod-lustre-osd-zfs module. That should force the installation of ZFS kernel modules, or report an error during kmod-lustre-osd-zfs installation if they are not available.

[Jian Yu]

Hi Andreas,
On my local RHEL 9.1 vm node, I found installing zfs required kmod-zfs. So, since "Requires: zfs" is already in lustre.spec.in, it seems we do not need to add "Requires: kmod-zfs".
I compared the node-provisioning logs on RHEL 8.7 zfs session and RHEL 9.1 zfs session on Maloo, and found:

RHEL 8.7

Yum installing zfs...

=========================================================================================================
 Package                                      Arch    Version                         Repository     Size
=========================================================================================================
Installing:
 zfs                                          x86_64  2.1.5-1.el8                     lustre-build  654 k
Installing dependencies:
 kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64   x86_64  2.1.5-1.el8                     lustre-build  1.5 M
 libnvpair3                                   x86_64  2.1.5-1.el8                     lustre-build   36 k
 libuutil3                                    x86_64  2.1.5-1.el8                     lustre-build   32 k
 libzfs5                                      x86_64  2.1.5-1.el8                     lustre-build  230 k
 libzpool5                                    x86_64  2.1.5-1.el8                     lustre-build  1.3 M
 lm_sensors-libs                              x86_64  3.4.0-23.20180522git70f7e08.el8 core-0         59 k
 sysstat                                      x86_64  11.7.3-7.el8                    core-1        425 k
<~snip~>
Installed:
  kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64-2.1.5-1.el8.x86_64                 
  libnvpair3-2.1.5-1.el8.x86_64                                                 
  libuutil3-2.1.5-1.el8.x86_64                                                  
  libzfs5-2.1.5-1.el8.x86_64                                                    
  libzpool5-2.1.5-1.el8.x86_64                                                  
  lm_sensors-libs-3.4.0-23.20180522git70f7e08.el8.x86_64                        
  sysstat-11.7.3-7.el8.x86_64                                                   
  zfs-2.1.5-1.el8.x86_64                                                        

Complete!                               

RHEL 9.1

Yum installing zfs...

================================================================================
 Package                Arch       Version               Repository        Size
================================================================================
Installing:
 zfs                    x86_64     2.1.5-1.el9           lustre-build     650 k
Installing dependencies:
 libuv                  x86_64     1:1.42.0-1.el9        core-1           153 k
 libzpool5              x86_64     2.1.5-1.el9           lustre-build     1.3 M
 lm_sensors-libs        x86_64     3.6.0-10.el9          core-1            44 k
 nss                    x86_64     3.79.0-14.el9_0       core-1           710 k
 nss-softokn            x86_64     3.79.0-14.el9_0       core-1           386 k
 nss-softokn-freebl     x86_64     3.79.0-14.el9_0       core-1           336 k
 nss-sysinit            x86_64     3.79.0-14.el9_0       core-1            24 k
 pcp-conf               x86_64     5.3.7-7.el9           core-1            34 k
 pcp-libs               x86_64     5.3.7-7.el9           core-1           607 k
 sysstat                x86_64     12.5.4-3.el9          core-1           484 k
<~snip~>
Package zfs-2.1.5-1.el9.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

I've no idea why installing zfs didn't depend on kmod-zfs on the RHEL 9.1 node provisioned by ljb, but it had such a dependency on RHEL 8.7 node and my local RHEL 9.1 vm node.

[Andreas Dilger]

You can run "rpm --requires zfs" and "rpm --suggests zfs". Maybe the el9 package only suggests kmod-zfs? Or the zfs.spec file for that build is broken somehow?

The spec in Git looks OK to me, and that line has been there for 10 years:

Requires:       %{name}-kmod = %{version}

Alternately, maybe the package was installed with "--nodeps"?

[Jian Yu]

On the el9.1 node provisioned by "ljb":

# rpm -qa | grep zfs
# yum install -y zfs
Updating Subscription Management repositories.
Last metadata expiration check: 3:50:41 ago on Thu 30 Mar 2023 01:25:47 PM UTC.
Dependencies resolved.
===========================================================================================================================================================================================================================================
 Package                                                 Architecture                                         Version                                                     Repository                                                  Size
===========================================================================================================================================================================================================================================
Installing:
 zfs                                                     x86_64                                               2.1.9-1.el9                                                 lustre-build                                               652 k
Installing dependencies:
 libzfs5                                                 x86_64                                               2.1.9-1.el9                                                 lustre-build                                               231 k
 libzpool5                                               x86_64                                               2.1.9-1.el9                                                 lustre-build                                               1.3 M
 zfs-dkms                                                noarch                                               2.1.9-1.el9                                                 lustre-build                                                30 M

Transaction Summary
===========================================================================================================================================================================================================================================
Install  4 Packages
<~snip~>
Installed:
  libzfs5-2.1.9-1.el9.x86_64                                libzpool5-2.1.9-1.el9.x86_64                                zfs-2.1.9-1.el9.x86_64                                zfs-dkms-2.1.9-1.el9.noarch                               

Complete!

# rpm -qa | grep kmod-zfs
# rpm -q --requires zfs | grep kmod
zfs-kmod = 2.1.9

[Andreas Dilger]

Jian, is it possible that kmod-zfs is already installed on this node, but for a different kernel version?

[Jian Yu]

Hi Andreas,
Before running "yum install -y zfs":

[root@trevis-86vm6 ~]# rpm -qa | grep zfs
[root@trevis-86vm6 ~]# rpm -qa | grep kmod
kmod-28-7.el9.x86_64
kmod-libs-28-7.el9.x86_64

After running "yum install -y zfs":

[root@trevis-86vm6 ~]# rpm -qa | grep zfs
libzfs5-2.1.9-1.el9.x86_64
zfs-dkms-2.1.9-1.el9.noarch
zfs-2.1.9-1.el9.x86_64
[root@trevis-86vm6 ~]# rpm -qa | grep kmod
kmod-28-7.el9.x86_64
kmod-libs-28-7.el9.x86_64

 

[Jian Yu]

I provisioned two test nodes for build https://build.whamcloud.com/job/lustre-reviews/93453/:

ljb install -d el8.7 -a x86_64 -p test -j lustre-reviews -b 93453 -t server -i inkernel -k vim -v -n trevis-87vm7
ljb install -d el9.1 -a x86_64 -p test -j lustre-reviews -b 93453 -t server -i inkernel -k vim -v -n trevis-86vm6

On el8.7 node trevis-87vm7:

[root@trevis-87vm7 ~]# rpm -qa | grep zfs

[root@trevis-87vm7 ~]# rpm -qa | grep kmod
kmod-lustre-osd-ldiskfs-2.15.54_140_gd739512-1.el8.x86_64
kmod-25-19.el8.x86_64  
kmod-lustre-2.15.54_140_gd739512-1.el8.x86_64
kmod-lustre-tests-2.15.54_140_gd739512-1.el8.x86_64
kmod-libs-25-19.el8.x86_64

[root@trevis-87vm7 ~]# yum deplist zfs
Updating Subscription Management repositories.
Last metadata expiration check: 0:18:21 ago on Sun 02 Apr 2023 04:37:12 AM UTC.
package: zfs-2.1.5-1.el8.x86_64
<~snip~>
  dependency: zfs-kmod = 2.1.5
   provider: kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64-2.1.5-1.el8.x86_64
   provider: zfs-dkms-2.1.5-1.el8.noarch

On el9.1 node trevis-86vm6:

[root@trevis-86vm6 ~]# rpm -qa | grep zfs

[root@trevis-86vm6 ~]# rpm -qa | grep kmod
kmod-28-7.el9.x86_64   
kmod-libs-28-7.el9.x86_64
kmod-lustre-2.15.54_140_gd739512-1.el9.x86_64
kmod-lustre-osd-ldiskfs-2.15.54_140_gd739512-1.el9.x86_64
kmod-lustre-tests-2.15.54_140_gd739512-1.el9.x86_64

[root@trevis-86vm6 ~]# yum deplist zfs
Updating Subscription Management repositories.
Last metadata expiration check: 0:19:53 ago on Sun 02 Apr 2023 04:35:23 AM UTC.
package: zfs-2.1.5-1.el9.x86_64
<~snip~>
  dependency: zfs-kmod = 2.1.5
   provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
   provider: zfs-dkms-2.1.5-1.el9.noarch

As we can see, on both nodes, zfs depends on zfs-kmod, which has two providers: kmod-zfs and zfs-dkms.

[Jian Yu]

However, while running "yum install -y zfs", kmod-zfs was installed on el8.7 node trevis-87vm7:

[root@trevis-87vm7 ~]# yum install -y zfs
Updating Subscription Management repositories.
Last metadata expiration check: 0:19:07 ago on Sun 02 Apr 2023 04:37:12 AM UTC.
Dependencies resolved.
========================================================================================================================================================================================================================================
 Package                                                                    Architecture                           Version                                                           Repository                                    Size
========================================================================================================================================================================================================================================
Installing:
 zfs                                                                        x86_64                                 2.1.5-1.el8                                                       lustre-build                                 654 k
Installing dependencies:
 kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64                                 x86_64                                 2.1.5-1.el8                                                       lustre-build                                 1.5 M
 libnvpair3                                                                 x86_64                                 2.1.5-1.el8                                                       lustre-build                                  36 k
 libuutil3                                                                  x86_64                                 2.1.5-1.el8                                                       lustre-build                                  32 k
 libzfs5                                                                    x86_64                                 2.1.5-1.el8                                                       lustre-build                                 230 k
 libzpool5                                                                  x86_64                                 2.1.5-1.el8                                                       lustre-build                                 1.3 M
 lm_sensors-libs                                                            x86_64                                 3.4.0-23.20180522git70f7e08.el8                                   core-0                                        59 k
 sysstat                                                                    x86_64                                 11.7.3-7.el8                                                      core-1                                       425 k

Transaction Summary
========================================================================================================================================================================================================================================
Install  8 Packages
<~snip~>
Installed:
  kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64-2.1.5-1.el8.x86_64           libnvpair3-2.1.5-1.el8.x86_64           libuutil3-2.1.5-1.el8.x86_64           libzfs5-2.1.5-1.el8.x86_64           libzpool5-2.1.5-1.el8.x86_64
  lm_sensors-libs-3.4.0-23.20180522git70f7e08.el8.x86_64                  sysstat-11.7.3-7.el8.x86_64             zfs-2.1.5-1.el8.x86_64

Complete!

but zfs-dkms instead of kmod-zfs was installed on el9.1 node trevis-86vm6:

[root@trevis-86vm6 ~]# yum install -y zfs
Updating Subscription Management repositories.
Last metadata expiration check: 0:21:27 ago on Sun 02 Apr 2023 04:35:23 AM UTC.
Dependencies resolved.
========================================================================================================================================================================================================================================
 Package                                                    Architecture                                Version                                                           Repository                                               Size
========================================================================================================================================================================================================================================
Installing:
 zfs                                                        x86_64                                      2.1.5-1.el9                                                       lustre-build                                            650 k
Installing dependencies:
 bison                                                      x86_64                                      3.7.4-5.el9                                                       core-1                                                  947 k
 dkms                                                       noarch                                      3.0.10-1.el9                                                      addon-epel9-x86_64                                       84 k
 elfutils-libelf-devel                                      x86_64                                      0.187-5.el9                                                       core-1                                                   27 k
 flex                                                       x86_64                                      2.6.4-9.el9                                                       core-1                                                  318 k
 kernel-devel                                               x86_64                                      5.14.0-162.18.1_lustre.el9                                        lustre-build                                             23 M
 kernel-devel-matched                                       x86_64                                      5.14.0-162.18.1_lustre.el9                                        lustre-build                                            1.9 M
 libnvpair3                                                 x86_64                                      2.1.5-1.el9                                                       lustre-build                                             39 k
 libuutil3                                                  x86_64                                      2.1.5-1.el9                                                       lustre-build                                             33 k
 libuv                                                      x86_64                                      1:1.42.0-1.el9                                                    core-1                                                  153 k
 libzfs5                                                    x86_64                                      2.1.5-1.el9                                                       lustre-build                                            231 k
 libzpool5                                                  x86_64                                      2.1.5-1.el9                                                       lustre-build                                            1.3 M
 lm_sensors-libs                                            x86_64                                      3.6.0-10.el9                                                      core-1                                                   44 k
 m4                                                         x86_64                                      1.4.19-1.el9                                                      core-1                                                  304 k
 nspr                                                       x86_64                                      4.34.0-14.el9_0                                                   core-1                                                  140 k
 nss                                                        x86_64                                      3.79.0-14.el9_0                                                   core-1                                                  710 k
 nss-softokn                                                x86_64                                      3.79.0-14.el9_0                                                   core-1                                                  386 k
 nss-softokn-freebl                                         x86_64                                      3.79.0-14.el9_0                                                   core-1                                                  336 k
 nss-sysinit                                                x86_64                                      3.79.0-14.el9_0                                                   core-1                                                   24 k
 nss-util                                                   x86_64                                      3.79.0-14.el9_0                                                   core-1                                                   92 k
 pcp-conf                                                   x86_64                                      5.3.7-7.el9                                                       core-1                                                   34 k
 pcp-libs                                                   x86_64                                      5.3.7-7.el9                                                       core-1                                                  607 k
 sysstat                                                    x86_64                                      12.5.4-3.el9                                                      core-1                                                  484 k
 zfs-dkms                                                   noarch                                      2.1.5-1.el9                                                       lustre-build                                             29 M

Transaction Summary
========================================================================================================================================================================================================================================
Install  24 Packages   
<~snip~>
Installed:
  bison-3.7.4-5.el9.x86_64                                 dkms-3.0.10-1.el9.noarch                    elfutils-libelf-devel-0.187-5.el9.x86_64   flex-2.6.4-9.el9.x86_64           kernel-devel-5.14.0-162.18.1_lustre.el9.x86_64
  kernel-devel-matched-5.14.0-162.18.1_lustre.el9.x86_64   libnvpair3-2.1.5-1.el9.x86_64               libuutil3-2.1.5-1.el9.x86_64               libuv-1:1.42.0-1.el9.x86_64       libzfs5-2.1.5-1.el9.x86_64
  libzpool5-2.1.5-1.el9.x86_64                             lm_sensors-libs-3.6.0-10.el9.x86_64         m4-1.4.19-1.el9.x86_64                     nspr-4.34.0-14.el9_0.x86_64       nss-3.79.0-14.el9_0.x86_64
  nss-softokn-3.79.0-14.el9_0.x86_64                       nss-softokn-freebl-3.79.0-14.el9_0.x86_64   nss-sysinit-3.79.0-14.el9_0.x86_64         nss-util-3.79.0-14.el9_0.x86_64   pcp-conf-5.3.7-7.el9.x86_64
  pcp-libs-5.3.7-7.el9.x86_64                              sysstat-12.5.4-3.el9.x86_64                 zfs-2.1.5-1.el9.x86_64                     zfs-dkms-2.1.5-1.el9.noarch

Complete!

[Jian Yu]

So, the issue now is why zfs-dkms instead of kmod-zfs was installed on el9.1 node.

[Jian Yu]

More info:

[root@trevis-87vm7 ~]# yum deplist /scratch/hudson/lustre-reviews/93453/arch-x86_64_build_type-server_distro-el8.7_ib_stack-inkernel/kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64-2.1.5-1.el8.x86_64.rpm | awk '/provider:/ {print $2}' | sort -u
Last metadata expiration check: 0:20:29 ago on Sun 02 Apr 2023 05:42:02 AM UTC.
bash-4.4.20-4.el8_6.x86_64
kernel-core-4.18.0-425.10.1.el8_lustre.x86_64
kernel-core-4.18.0-425.13.1.el8_7.x86_64
kernel-debug-core-4.18.0-425.13.1.el8_7.x86_64
kmod-25-19.el8.x86_64
zfs-2.1.5-1.el8.x86_64

[root@trevis-86vm6 ~]# yum deplist /scratch/hudson/lustre-reviews/93453/arch-x86_64_build_type-server_distro-el9.1_ib_stack-inkernel/kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm
<~snip~>
  dependency: ksym(EdonRFinal) = 0xd345f388
   provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
   provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
  dependency: ksym(EdonRHash) = 0xe7e4950f
   provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
   provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
<~snip~>

[root@trevis-86vm6 ~]# yum deplist /scratch/hudson/lustre-reviews/93453/arch-x86_64_build_type-server_distro-el9.1_ib_stack-inkernel/kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm | awk '/provider:/ {print $2}' | sort -u
Last metadata expiration check: 0:53:54 ago on Sun 02 Apr 2023 05:02:55 AM UTC.
bash-5.1.8-6.el9_1.x86_64
kernel-core-5.14.0-162.18.1_lustre.el9.x86_64
kmod-28-7.el9.x86_64
kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
zfs-2.1.5-1.el9.x86_64

On el9.1 node, installing kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm depends on the symbols provided by itself.

[Andreas Dilger]

I think in newer RPM there is a way to say that kmod-zfs is preferred over zfs-dkms, but this would probably be a change in the zfs.spec itself and would only work for ZFS that we build ourselves?

[Jian Yu]

Hi adilger,
I found even installing kmod-zfs first, it also failed on the el9.1 node provisioned by "ljb":

[root@trevis-86vm6 ~]# yum install -y kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
Updating Subscription Management repositories.
Last metadata expiration check: 3:14:21 ago on Mon 03 Apr 2023 06:13:33 PM UTC.
Error:
 Problem: conflicting requests
  - nothing provides ksym(__SCT__cond_resched) = 0xc07351b3 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
  - nothing provides ksym(__SCT__might_resched) = 0xe2c17b5d needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
  - nothing provides ksym(__SCT__preempt_schedule) = 0x48d88a2c needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
  - nothing provides ksym(__alloc_pages) = 0x26458776 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
  - nothing provides ksym(__alloc_percpu) = 0x949f7342 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
  - nothing provides ksym(__bitmap_weight) = 0x63c4d61f needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
<~snip~>

I didn't hit the above issue on my local VirtualBox RHEL 9.1 vm node. Manually building, installing (zfs and Lustre) and running basic sanity tests with FSTYPE=zfs passed.

[Jian Yu]

Since Jenkins build system uses lbuild to build packages, we have to investigate the lbuild codes, which needs more time.
Hi mdiep, do you have any advice?

[Jian Yu]

The steps for manually building and installation of zfs are straightforward:

# git clone https://github.com/zfsonlinux/zfs.git
# cd zfs
# git co zfs-2.1.5
# git switch -c zfs-2.1.5

# sh autogen.sh
# ./configure --with-spec=redhat --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.12.1.el9_1/linux-5.14.0-162.12.1_lustre.el9.x86_64
# make rpms

# rpm -ivh zfs-2.1.5-1.el9.x86_64.rpm kmod-zfs-2.1.5-1.el9.x86_64.rpm kmod-zfs-devel-2.1.5-1.el9.x86_64.rpm libzfs5-2.1.5-1.el9.x86_64.rpm libzfs5-devel-2.1.5-1.el9.x86_64.rpm libzpool5-2.1.5-1.el9.x86_64.rpm libnvpair3-2.1.5-1.el9.x86_64.rpm libuutil3-2.1.5-1.el9.x86_64.rpm

However, in lbuild:build_spl_zfs(), the steps are more complicated:

+ git clone -n /data/gerrit_mirror/fs/zfs.git zfs
+ git checkout -b lbuild zfs-2.1.5

+ bash autogen.sh
+ ./configure --with-linux=/var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/reused/usr/src/kernels/5.14.0-162.22.2_lustre.el9.x86_64 --with-linux-obj=/var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/reused/usr/src/kernels/5.14.0-162.22.2_lustre.el9.x86_64
+ make dist
+ ln -f zfs/zfs-2.1.5.tar.gz /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/SOURCES
+ ln -f zfs/scripts/kmodtool /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/SOURCES/

+ rpmbuild -ba zfs/rpm/generic/zfs.spec --nodeps ...
+ rpmbuild -ba zfs/rpm/generic/zfs-kmod.spec --nodeps ...
+ rpmbuild -ba zfs/rpm/generic/zfs-dkms.spec --nodeps ......

+ rpm2cpio /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/RPMS/x86_64/kmod-zfs-devel-2.1.5-1.el9.x86_64.rpm
+ cpio -id

+ CONFIGURE_FLAGS='--with-zfs-obj=...... --with-zfs=...... --with-spl-obj=...... --with-spl=...... --with-zfs-devel=...... '
+ OSDZFSRPM=true
+ return 0

[Jian Yu]

In order to debug lbuild build_spl_zfs(), I performed the commands step by step as follows on my local RHEL 9.1 vm node but found kmod-zfs rpm could be successfully built and installed:

# rpm -qa | grep zfs
# mkdir /root/BUILD
# cd BUILD/
# mkdir -p RPMS RPMS/x86_64 RPMS/noarch BUILD SOURCES SPECS SRPMS
# cd

# git clone https://github.com/zfsonlinux/zfs.git
# cd zfs
# git checkout -b lbuild zfs-2.1.5
# bash autogen.sh
# ./configure --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64 --with-linux-obj=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64
# make dist
# cd ../
# ln -f zfs/zfs-2.1.5.tar.gz /root/BUILD/SOURCES
# ln -f zfs/scripts/kmodtool /root/BUILD/SOURCES

# rpmbuild -ba zfs/rpm/generic/zfs.spec --nodeps -v --define '_use_internal_dependency_generator 0' --define 'require_kdir /root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64' --define 'kver 5.14.0-162.22.2_lustre.el9.x86_64' --define 'kernels 5.14.0-162.22.2_lustre.el9.x86_64' --define '_tmppath /var/tmp' --define 'kernelbuildroot /' --define '_topdir /root/BUILD'

# rpmbuild -ba zfs/rpm/generic/zfs-kmod.spec --nodeps -v --define '_use_internal_dependency_generator 0' --define 'require_kdir /root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64' --define 'kver 5.14.0-162.22.2_lustre.el9.x86_64' --define 'kernels 5.14.0-162.22.2_lustre.el9.x86_64' --define '_tmppath /var/tmp' --define 'kernelbuildroot /' --define '_topdir /root/BUILD'

# rpmbuild -ba zfs/rpm/generic/zfs-dkms.spec --nodeps -v --define '_use_internal_dependency_generator 0' --define 'require_kdir /root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64' --define 'kver 5.14.0-162.22.2_lustre.el9.x86_64' --define 'kernels 5.14.0-162.22.2_lustre.el9.x86_64' --define '_tmppath /var/tmp' --define 'kernelbuildroot /' --define '_topdir /root/BUILD'

# cd /root/BUILD/RPMS/x86_64/
# rpm -ivh zfs-2.1.5-1.el9.x86_64.rpm kmod-zfs-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm kmod-zfs-devel-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm libzfs5-2.1.5-1.el9.x86_64.rpm libzfs5-devel-2.1.5-1.el9.x86_64.rpm libzpool5-2.1.5-1.el9.x86_64.rpm libnvpair3-2.1.5-1.el9.x86_64.rpm libuutil3-2.1.5-1.el9.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:libnvpair3-2.1.5-1.el9           ################################# [ 13%]
   2:libuutil3-2.1.5-1.el9            ################################# [ 25%]
   3:libzfs5-2.1.5-1.el9              ################################# [ 38%]
   4:libzpool5-2.1.5-1.el9            ################################# [ 50%]
   5:kmod-zfs-5.14.0-162.22.2_lustre.e################################# [ 63%]
   6:zfs-2.1.5-1.el9                  ################################# [ 75%]
Created symlink /etc/systemd/system/zfs-import.target.wants/zfs-import-cache.service → /usr/lib/systemd/system/zfs-import-cache.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-mount.service → /usr/lib/systemd/system/zfs-mount.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-share.service → /usr/lib/systemd/system/zfs-share.service.
Created symlink /etc/systemd/system/zed.service → /usr/lib/systemd/system/zfs-zed.service.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-zed.service → /usr/lib/systemd/system/zfs-zed.service.
Created symlink /etc/systemd/system/multi-user.target.wants/zfs.target → /usr/lib/systemd/system/zfs.target.
Created symlink /etc/systemd/system/zfs.target.wants/zfs-import.target → /usr/lib/systemd/system/zfs-import.target.
Created symlink /etc/systemd/system/zfs-volumes.target.wants/zfs-volume-wait.service → /usr/lib/systemd/system/zfs-volume-wait.service.
   7:libzfs5-devel-2.1.5-1.el9        ################################# [ 88%]
   8:kmod-zfs-devel-5.14.0-162.22.2_lu################################# [100%]

# rpm -qa | grep kmod-zfs
kmod-zfs-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64
kmod-zfs-devel-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64

 

[Jian Yu]

The issue was narrowed down to Jenkins build system and/or ljb tool.
Hi mdiep and mkvardakov, could you please investigate further? Thank you.

[Minh Diep]

colmstea, could you check ljb tool?

[Charlie Olmstead]

This does not look like an LJB issue. LJB is installing the kernel from the build, setting the proper default kernel and rebooting the VM to apply the kernel change (Jian's build from above no longer exists so I used https://build.whamcloud.com/view/Reviews/job/lustre-reviews/94201/)

[root@onyx-22vm1 ~]# uname -a
Linux onyx-22vm1.onyx.whamcloud.com 5.14.0-162.18.1_lustre.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 11 20:39:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
 

LJB installed the expected repo

[root@onyx-22vm1 ~]# cat /etc/yum.repos.d/lustre-build.repo
[lustre-build]
name=lustre-build
baseurl=http://mgmt/cobbler/localmirror/hudson/lustre-reviews/94201/arch-x86_64_build_type-server_distro-el9.1_ib_stack-inkernel
enabled=1
priority=1
gpgcheck=0 

 

When I install zfs, yum installs zfs-dkms as Jian saw. If I install kmod-zfs I get lots of 'nothing provides ksym' errors as Jian saw also.

...
  - nothing provides ksym(__SCT__cond_resched) = 0xc07351b3 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
  - nothing provides ksym(__SCT__might_resched) = 0xe2c17b5d needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
  - nothing provides ksym(__SCT__preempt_schedule) = 0x48d88a2c needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64
...

 

I looked at the console log for the build of the el9.1 server and see a lot of messages like this. Perhaps this has something to do with it?

11:47:46 ********************************************************************************
11:47:46 *********************** KERNEL ABI COMPATIBILITY WARNING ***********************
11:47:46 ********************************************************************************
11:47:46 The following kernel symbols are not guaranteed to remain compatible with
11:47:46 future kernel updates to this RHEL release:
11:47:46 
11:47:46 	__SCT__cond_resched
11:47:46 	__SCT__preempt_schedule_notrace
11:47:46 	___ratelimit
11:47:46 	__bdev_dax_supported 

 

[Gerrit Updater]

"Minh Diep <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/50746
Subject: LU-16657 build: don't need to use zfs kmodtool on RHEL9.1
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 21bb46e6b7bb8d392fda8cec5aaa25304eed18f8

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/50746/
Subject: LU-16657 build: don't need to use zfs kmodtool on RHEL9.1
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 8a967f69e1f7f3e5b72766ae47db3bf64c76bac0

[Peter Jones]

Landed for 2.16