https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16657 Lustre <p>RHEL 9.1 zfs test session failed:<br/> <a href="https://testing.whamcloud.com/test_sessions/1dd26fcc-a33f-498f-a2a9-8eb743911601" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/1dd26fcc-a33f-498f-a2a9-8eb743911601</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2023-02-14T06:43:02 onyx-70vm4 - zfs successfully installed 2023-02-14T06:43:02 onyx-70vm4 - Yum installing kmod-lustre-osd-zfs... 2023-02-14T06:43:02 onyx-70vm4 - Attempt 1 2023-02-14T06:43:04 onyx-70vm3 - yum install completed: out=[Updating Subscription Management repositories. Last metadata expiration check: 0:12:27 ago on Tue 14 Feb 2023 06:30:36 AM UTC. (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) ] err=[Error: Problem: package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(__cv_broadcast) = 0xc93ea281, but none of the providers can be installed - package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(arc_add_prune_callback) = 0x1363912f, but none of the providers can be installed - package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(arc_buf_size) = 0x4f87e6b0, but none of the providers can be installed - package kmod-lustre-osd-zfs-2.15.53_154_g87af414-1.el9.x86_64 requires ksym(arc_remove_prune_callback) = 0x1ab2d851, but none of the providers can be installed --------8&lt;-------- </pre> </div></div> LU-16657

installing kmod-lustre-osd-zfs failed for RHEL 9.1 server

Bug Blocker Resolved Fixed Minh Diep Jian Yu Tue, 14 Feb 2023 20:47:25 +0000 Thu, 4 May 2023 23:11:39 +0000 Mon, 1 May 2023 06:37:47 +0000 Lustre 2.16.0 0 8 <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=yujian" class="user-hover" rel="yujian">yujian</a> I could not reproduce the issue. Could restart the build, please?</p> <p>Sure, Michael. I just retriggered the failed test session in <a href="https://review.whamcloud.com/49921" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/49921</a>.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mkvardakov" class="user-hover" rel="mkvardakov">mkvardakov</a> did the re-run test get you what you need to know?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=pjones" class="user-hover" rel="pjones">pjones</a> not much. The issue is duplicate of <a href="https://jira.whamcloud.com/browse/LU-16006" title="can&#39;t install kmod-lustre-osd-zfs" class="issue-link" data-issue-key="LU-16006">LU-16006</a> which is fixed, I guess. Jian rebased the code but that did not help.</p> <p>has this ever worked before? this is new rhel9.1 server support</p> <p>I took a quick look at the build log. I don't think this is DCO issue. it will require a large amount of time for me to dig into this</p> <p>I update kernel-devel, kernel... on the builder el9.1 but still not helping. Unless this works in manual build, I'd say this is not DCO issue. <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=yujian" class="user-hover" rel="yujian">yujian</a> please open LU ticket so <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=cfaber" class="user-hover" rel="cfaber">cfaber</a> can assign someone to look into this further.</p> <p>Thank you, Minh. Let me try to build and install manually.</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mdiep" class="user-hover" rel="mdiep">mdiep</a>,<br/> After building zfs, I got the following rpms:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls *.x86_64.rpm | grep -v debug kmod-zfs-2.1.9-1.el9.x86_64.rpm kmod-zfs-devel-2.1.9-1.el9.x86_64.rpm libnvpair3-2.1.9-1.el9.x86_64.rpm libuutil3-2.1.9-1.el9.x86_64.rpm libzfs5-2.1.9-1.el9.x86_64.rpm libzfs5-devel-2.1.9-1.el9.x86_64.rpm libzpool5-2.1.9-1.el9.x86_64.rpm zfs-2.1.9-1.el9.x86_64.rpm zfs-test-2.1.9-1.el9.x86_64.rpm </pre> </div></div> <p>And after installing them, I can successfully install kmod-lustre-osd-zfs. Could you please check if any zfs related rpms were not installed while provisioning the node?</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=yujian" class="user-hover" rel="yujian">yujian</a> </p> <p>Could you tell me the steps you did? I'd like to compare with lbuild. there must be something new that we missed</p> <p>After building and installing the patched kernel, I built zfs as follows and installed the rpms listed in the previous comment:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># sh autogen.sh # ./configure --with-spec=redhat --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.12.1.el9_1/linux-5.14.0-162.12.1_lustre.el9.x86_64 # make rpms </pre> </div></div> <p>Then I built Lustre as follows and installed the rpms successfully:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># sh autogen.sh # ./configure --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.12.1.el9_1/linux-5.14.0-162.12.1_lustre.el9.x86_64 --with-zfs=/root/zfs --without-ldiskfs # make rpms </pre> </div></div> <p>but this is not the same kernel as in the patch. could you try on </p> <p>5.14.0-162.18.1_lustre.el9.x86_64</p> <p>I already have a built kernel 5.14.0-162.12.1 on my local RHEL 9.1 vm node, so I didn't take another 4 hours to build kernel 5.14.0-162.18.1.<br/> IMHO, there will be no problem for it.</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mdiep" class="user-hover" rel="mdiep">mdiep</a>,<br/> I ran the following command to provision an el9.1 server node:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ljb install -d el9.1 -a x86_64 -p test -j lustre-reviews -b 93153 -t server -i inkernel -v -n trevis-86vm6 </pre> </div></div> <p>As per the node-provisioning log on Maloo, I ran "yum install -y zfs" on the node. Then I found <b>kmod-zfs</b> rpm was not installed, which is the root cause of this ticket.</p> <p>Jian, can you please push a patch that adds "<tt>Requires: kmod-zfs</tt>" (or possibly "<tt>Requires: kmp-zfs</tt>" for SLES) to the <tt>kmod-lustre-osd-zfs</tt> module. That should force the installation of ZFS kernel modules, or report an error during <tt>kmod-lustre-osd-zfs</tt> installation if they are not available.</p> <p>Hi Andreas,<br/> On my local RHEL 9.1 vm node, I found installing <tt>zfs</tt> required <tt>kmod-zfs</tt>. So, since "<tt>Requires: zfs</tt>" is already in lustre.spec.in, it seems we do not need to add "<tt>Requires: kmod-zfs</tt>".<br/> I compared the node-provisioning logs on RHEL 8.7 zfs session and RHEL 9.1 zfs session on Maloo, and found:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>RHEL 8.7</b></div><div class="preformattedContent panelContent"> <pre>Yum installing zfs... ========================================================================================================= Package Arch Version Repository Size ========================================================================================================= Installing: zfs x86_64 2.1.5-1.el8 lustre-build 654 k Installing dependencies: kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64 x86_64 2.1.5-1.el8 lustre-build 1.5 M libnvpair3 x86_64 2.1.5-1.el8 lustre-build 36 k libuutil3 x86_64 2.1.5-1.el8 lustre-build 32 k libzfs5 x86_64 2.1.5-1.el8 lustre-build 230 k libzpool5 x86_64 2.1.5-1.el8 lustre-build 1.3 M lm_sensors-libs x86_64 3.4.0-23.20180522git70f7e08.el8 core-0 59 k sysstat x86_64 11.7.3-7.el8 core-1 425 k &lt;~snip~&gt; Installed: kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64-2.1.5-1.el8.x86_64 libnvpair3-2.1.5-1.el8.x86_64 libuutil3-2.1.5-1.el8.x86_64 libzfs5-2.1.5-1.el8.x86_64 libzpool5-2.1.5-1.el8.x86_64 lm_sensors-libs-3.4.0-23.20180522git70f7e08.el8.x86_64 sysstat-11.7.3-7.el8.x86_64 zfs-2.1.5-1.el8.x86_64 Complete! </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>RHEL 9.1</b></div><div class="preformattedContent panelContent"> <pre>Yum installing zfs... ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: zfs x86_64 2.1.5-1.el9 lustre-build 650 k Installing dependencies: libuv x86_64 1:1.42.0-1.el9 core-1 153 k libzpool5 x86_64 2.1.5-1.el9 lustre-build 1.3 M lm_sensors-libs x86_64 3.6.0-10.el9 core-1 44 k nss x86_64 3.79.0-14.el9_0 core-1 710 k nss-softokn x86_64 3.79.0-14.el9_0 core-1 386 k nss-softokn-freebl x86_64 3.79.0-14.el9_0 core-1 336 k nss-sysinit x86_64 3.79.0-14.el9_0 core-1 24 k pcp-conf x86_64 5.3.7-7.el9 core-1 34 k pcp-libs x86_64 5.3.7-7.el9 core-1 607 k sysstat x86_64 12.5.4-3.el9 core-1 484 k &lt;~snip~&gt; Package zfs-2.1.5-1.el9.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! </pre> </div></div> <p>I've no idea why installing <tt>zfs</tt> didn't depend on <tt>kmod-zfs</tt> on the RHEL 9.1 node provisioned by <tt>ljb</tt>, but it had such a dependency on RHEL 8.7 node and my local RHEL 9.1 vm node.</p> <p>You can run "<tt>rpm --requires zfs</tt>" and "<tt>rpm --suggests zfs</tt>". Maybe the el9 package only suggests kmod-zfs? Or the zfs.spec file for that build is broken somehow?</p> <p>The spec in Git looks OK to me, and that line has been there for 10 years:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Requires: %{name}-kmod = %{version} </pre> </div></div> <p>Alternately, maybe the package was installed with "<tt>&#45;&#45;nodeps</tt>"?</p> <p>On the el9.1 node provisioned by "ljb":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># rpm -qa | grep zfs # yum install -y zfs Updating Subscription Management repositories. Last metadata expiration check: 3:50:41 ago on Thu 30 Mar 2023 01:25:47 PM UTC. Dependencies resolved. =========================================================================================================================================================================================================================================== Package Architecture Version Repository Size =========================================================================================================================================================================================================================================== Installing: zfs x86_64 2.1.9-1.el9 lustre-build 652 k Installing dependencies: libzfs5 x86_64 2.1.9-1.el9 lustre-build 231 k libzpool5 x86_64 2.1.9-1.el9 lustre-build 1.3 M zfs-dkms noarch 2.1.9-1.el9 lustre-build 30 M Transaction Summary =========================================================================================================================================================================================================================================== Install 4 Packages &lt;~snip~&gt; Installed: libzfs5-2.1.9-1.el9.x86_64 libzpool5-2.1.9-1.el9.x86_64 zfs-2.1.9-1.el9.x86_64 zfs-dkms-2.1.9-1.el9.noarch Complete! </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># rpm -qa | grep kmod-zfs # rpm -q --requires zfs | grep kmod zfs-kmod = 2.1.9 </pre> </div></div> <p>Jian, is it possible that <tt>kmod-zfs</tt> is already installed on this node, but for a different kernel version?</p> <p>Hi Andreas,<br/> Before running "yum install -y zfs":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@trevis-86vm6 ~]# rpm -qa | grep zfs [root@trevis-86vm6 ~]# rpm -qa | grep kmod kmod-28-7.el9.x86_64 kmod-libs-28-7.el9.x86_64 </pre> </div></div> <p>After running "yum install -y zfs":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@trevis-86vm6 ~]# rpm -qa | grep zfs libzfs5-2.1.9-1.el9.x86_64 zfs-dkms-2.1.9-1.el9.noarch zfs-2.1.9-1.el9.x86_64 [root@trevis-86vm6 ~]# rpm -qa | grep kmod kmod-28-7.el9.x86_64 kmod-libs-28-7.el9.x86_64 </pre> </div></div> <p> </p> <p>I provisioned two test nodes for build <a href="https://build.whamcloud.com/job/lustre-reviews/93453/:" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-reviews/93453/:</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ljb install -d el8.7 -a x86_64 -p test -j lustre-reviews -b 93453 -t server -i inkernel -k vim -v -n trevis-87vm7 ljb install -d el9.1 -a x86_64 -p test -j lustre-reviews -b 93453 -t server -i inkernel -k vim -v -n trevis-86vm6 </pre> </div></div> <p>On el8.7 node trevis-87vm7:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@trevis-87vm7 ~]# rpm -qa | grep zfs [root@trevis-87vm7 ~]# rpm -qa | grep kmod kmod-lustre-osd-ldiskfs-2.15.54_140_gd739512-1.el8.x86_64 kmod-25-19.el8.x86_64 kmod-lustre-2.15.54_140_gd739512-1.el8.x86_64 kmod-lustre-tests-2.15.54_140_gd739512-1.el8.x86_64 kmod-libs-25-19.el8.x86_64 [root@trevis-87vm7 ~]# yum deplist zfs Updating Subscription Management repositories. Last metadata expiration check: 0:18:21 ago on Sun 02 Apr 2023 04:37:12 AM UTC. package: zfs-2.1.5-1.el8.x86_64 &lt;~snip~&gt; dependency: zfs-kmod = 2.1.5 provider: kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64-2.1.5-1.el8.x86_64 provider: zfs-dkms-2.1.5-1.el8.noarch </pre> </div></div> <p>On el9.1 node trevis-86vm6:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@trevis-86vm6 ~]# rpm -qa | grep zfs [root@trevis-86vm6 ~]# rpm -qa | grep kmod kmod-28-7.el9.x86_64 kmod-libs-28-7.el9.x86_64 kmod-lustre-2.15.54_140_gd739512-1.el9.x86_64 kmod-lustre-osd-ldiskfs-2.15.54_140_gd739512-1.el9.x86_64 kmod-lustre-tests-2.15.54_140_gd739512-1.el9.x86_64 [root@trevis-86vm6 ~]# yum deplist zfs Updating Subscription Management repositories. Last metadata expiration check: 0:19:53 ago on Sun 02 Apr 2023 04:35:23 AM UTC. package: zfs-2.1.5-1.el9.x86_64 &lt;~snip~&gt; dependency: zfs-kmod = 2.1.5 provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 provider: zfs-dkms-2.1.5-1.el9.noarch </pre> </div></div> <p>As we can see, on both nodes, zfs depends on zfs-kmod, which has two providers: <b>kmod-zfs</b> and <b>zfs-dkms</b>.</p> <p>However, while running "yum install -y zfs", <b>kmod-zfs</b> was installed on el8.7 node trevis-87vm7:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@trevis-87vm7 ~]# yum install -y zfs Updating Subscription Management repositories. Last metadata expiration check: 0:19:07 ago on Sun 02 Apr 2023 04:37:12 AM UTC. Dependencies resolved. ======================================================================================================================================================================================================================================== Package Architecture Version Repository Size ======================================================================================================================================================================================================================================== Installing: zfs x86_64 2.1.5-1.el8 lustre-build 654 k Installing dependencies: kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64 x86_64 2.1.5-1.el8 lustre-build 1.5 M libnvpair3 x86_64 2.1.5-1.el8 lustre-build 36 k libuutil3 x86_64 2.1.5-1.el8 lustre-build 32 k libzfs5 x86_64 2.1.5-1.el8 lustre-build 230 k libzpool5 x86_64 2.1.5-1.el8 lustre-build 1.3 M lm_sensors-libs x86_64 3.4.0-23.20180522git70f7e08.el8 core-0 59 k sysstat x86_64 11.7.3-7.el8 core-1 425 k Transaction Summary ======================================================================================================================================================================================================================================== Install 8 Packages &lt;~snip~&gt; Installed: kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64-2.1.5-1.el8.x86_64 libnvpair3-2.1.5-1.el8.x86_64 libuutil3-2.1.5-1.el8.x86_64 libzfs5-2.1.5-1.el8.x86_64 libzpool5-2.1.5-1.el8.x86_64 lm_sensors-libs-3.4.0-23.20180522git70f7e08.el8.x86_64 sysstat-11.7.3-7.el8.x86_64 zfs-2.1.5-1.el8.x86_64 Complete! </pre> </div></div> <p>but <b>zfs-dkms</b> instead of kmod-zfs was installed on el9.1 node trevis-86vm6:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@trevis-86vm6 ~]# yum install -y zfs Updating Subscription Management repositories. Last metadata expiration check: 0:21:27 ago on Sun 02 Apr 2023 04:35:23 AM UTC. Dependencies resolved. ======================================================================================================================================================================================================================================== Package Architecture Version Repository Size ======================================================================================================================================================================================================================================== Installing: zfs x86_64 2.1.5-1.el9 lustre-build 650 k Installing dependencies: bison x86_64 3.7.4-5.el9 core-1 947 k dkms noarch 3.0.10-1.el9 addon-epel9-x86_64 84 k elfutils-libelf-devel x86_64 0.187-5.el9 core-1 27 k flex x86_64 2.6.4-9.el9 core-1 318 k kernel-devel x86_64 5.14.0-162.18.1_lustre.el9 lustre-build 23 M kernel-devel-matched x86_64 5.14.0-162.18.1_lustre.el9 lustre-build 1.9 M libnvpair3 x86_64 2.1.5-1.el9 lustre-build 39 k libuutil3 x86_64 2.1.5-1.el9 lustre-build 33 k libuv x86_64 1:1.42.0-1.el9 core-1 153 k libzfs5 x86_64 2.1.5-1.el9 lustre-build 231 k libzpool5 x86_64 2.1.5-1.el9 lustre-build 1.3 M lm_sensors-libs x86_64 3.6.0-10.el9 core-1 44 k m4 x86_64 1.4.19-1.el9 core-1 304 k nspr x86_64 4.34.0-14.el9_0 core-1 140 k nss x86_64 3.79.0-14.el9_0 core-1 710 k nss-softokn x86_64 3.79.0-14.el9_0 core-1 386 k nss-softokn-freebl x86_64 3.79.0-14.el9_0 core-1 336 k nss-sysinit x86_64 3.79.0-14.el9_0 core-1 24 k nss-util x86_64 3.79.0-14.el9_0 core-1 92 k pcp-conf x86_64 5.3.7-7.el9 core-1 34 k pcp-libs x86_64 5.3.7-7.el9 core-1 607 k sysstat x86_64 12.5.4-3.el9 core-1 484 k zfs-dkms noarch 2.1.5-1.el9 lustre-build 29 M Transaction Summary ======================================================================================================================================================================================================================================== Install 24 Packages &lt;~snip~&gt; Installed: bison-3.7.4-5.el9.x86_64 dkms-3.0.10-1.el9.noarch elfutils-libelf-devel-0.187-5.el9.x86_64 flex-2.6.4-9.el9.x86_64 kernel-devel-5.14.0-162.18.1_lustre.el9.x86_64 kernel-devel-matched-5.14.0-162.18.1_lustre.el9.x86_64 libnvpair3-2.1.5-1.el9.x86_64 libuutil3-2.1.5-1.el9.x86_64 libuv-1:1.42.0-1.el9.x86_64 libzfs5-2.1.5-1.el9.x86_64 libzpool5-2.1.5-1.el9.x86_64 lm_sensors-libs-3.6.0-10.el9.x86_64 m4-1.4.19-1.el9.x86_64 nspr-4.34.0-14.el9_0.x86_64 nss-3.79.0-14.el9_0.x86_64 nss-softokn-3.79.0-14.el9_0.x86_64 nss-softokn-freebl-3.79.0-14.el9_0.x86_64 nss-sysinit-3.79.0-14.el9_0.x86_64 nss-util-3.79.0-14.el9_0.x86_64 pcp-conf-5.3.7-7.el9.x86_64 pcp-libs-5.3.7-7.el9.x86_64 sysstat-12.5.4-3.el9.x86_64 zfs-2.1.5-1.el9.x86_64 zfs-dkms-2.1.5-1.el9.noarch Complete! </pre> </div></div> <p>So, the issue now is why <b>zfs-dkms</b> instead of <b>kmod-zfs</b> was installed on el9.1 node.</p> <p>More info:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@trevis-87vm7 ~]# yum deplist /scratch/hudson/lustre-reviews/93453/arch-x86_64_build_type-server_distro-el8.7_ib_stack-inkernel/kmod-zfs-4.18.0-425.10.1.el8_lustre.x86_64-2.1.5-1.el8.x86_64.rpm | awk '/provider:/ {print $2}' | sort -u Last metadata expiration check: 0:20:29 ago on Sun 02 Apr 2023 05:42:02 AM UTC. bash-4.4.20-4.el8_6.x86_64 kernel-core-4.18.0-425.10.1.el8_lustre.x86_64 kernel-core-4.18.0-425.13.1.el8_7.x86_64 kernel-debug-core-4.18.0-425.13.1.el8_7.x86_64 kmod-25-19.el8.x86_64 zfs-2.1.5-1.el8.x86_64 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@trevis-86vm6 ~]# yum deplist /scratch/hudson/lustre-reviews/93453/arch-x86_64_build_type-server_distro-el9.1_ib_stack-inkernel/kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm &lt;~snip~&gt; dependency: ksym(EdonRFinal) = 0xd345f388 provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 dependency: ksym(EdonRHash) = 0xe7e4950f provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 provider: kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 &lt;~snip~&gt; [root@trevis-86vm6 ~]# yum deplist /scratch/hudson/lustre-reviews/93453/arch-x86_64_build_type-server_distro-el9.1_ib_stack-inkernel/kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm | awk '/provider:/ {print $2}' | sort -u Last metadata expiration check: 0:53:54 ago on Sun 02 Apr 2023 05:02:55 AM UTC. bash-5.1.8-6.el9_1.x86_64 kernel-core-5.14.0-162.18.1_lustre.el9.x86_64 kmod-28-7.el9.x86_64 kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 zfs-2.1.5-1.el9.x86_64 </pre> </div></div> <p>On el9.1 node, installing kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm depends on the symbols provided by itself.</p> <p>I think in newer RPM there is a way to say that kmod-zfs is preferred over zfs-dkms, but this would probably be a change in the zfs.spec itself and would only work for ZFS that we build ourselves? </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>,<br/> I found even installing kmod-zfs first, it also failed on the el9.1 node provisioned by "ljb":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@trevis-86vm6 ~]# yum install -y kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 Updating Subscription Management repositories. Last metadata expiration check: 3:14:21 ago on Mon 03 Apr 2023 06:13:33 PM UTC. Error: Problem: conflicting requests - nothing provides ksym(__SCT__cond_resched) = 0xc07351b3 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__SCT__might_resched) = 0xe2c17b5d needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__SCT__preempt_schedule) = 0x48d88a2c needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__alloc_pages) = 0x26458776 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__alloc_percpu) = 0x949f7342 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 - nothing provides ksym(__bitmap_weight) = 0x63c4d61f needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 &lt;~snip~&gt; </pre> </div></div> <p>I didn't hit the above issue on my local VirtualBox RHEL 9.1 vm node. Manually building, installing (zfs and Lustre) and running basic sanity tests with FSTYPE=zfs passed.</p> <p>Since Jenkins build system uses lbuild to build packages, we have to investigate the lbuild codes, which needs more time.<br/> Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mdiep" class="user-hover" rel="mdiep">mdiep</a>, do you have any advice?</p> <p>The steps for manually building and installation of zfs are straightforward:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># git clone https://github.com/zfsonlinux/zfs.git # cd zfs # git co zfs-2.1.5 # git switch -c zfs-2.1.5 # sh autogen.sh # ./configure --with-spec=redhat --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.12.1.el9_1/linux-5.14.0-162.12.1_lustre.el9.x86_64 # make rpms # rpm -ivh zfs-2.1.5-1.el9.x86_64.rpm kmod-zfs-2.1.5-1.el9.x86_64.rpm kmod-zfs-devel-2.1.5-1.el9.x86_64.rpm libzfs5-2.1.5-1.el9.x86_64.rpm libzfs5-devel-2.1.5-1.el9.x86_64.rpm libzpool5-2.1.5-1.el9.x86_64.rpm libnvpair3-2.1.5-1.el9.x86_64.rpm libuutil3-2.1.5-1.el9.x86_64.rpm </pre> </div></div> <p>However, in <tt>lbuild:build_spl_zfs()</tt>, the steps are more complicated:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>+ git clone -n /data/gerrit_mirror/fs/zfs.git zfs + git checkout -b lbuild zfs-2.1.5 + bash autogen.sh + ./configure --with-linux=/var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/reused/usr/src/kernels/5.14.0-162.22.2_lustre.el9.x86_64 --with-linux-obj=/var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/reused/usr/src/kernels/5.14.0-162.22.2_lustre.el9.x86_64 + make dist + ln -f zfs/zfs-2.1.5.tar.gz /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/SOURCES + ln -f zfs/scripts/kmodtool /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/SOURCES/ + rpmbuild -ba zfs/rpm/generic/zfs.spec --nodeps ... + rpmbuild -ba zfs/rpm/generic/zfs-kmod.spec --nodeps ... + rpmbuild -ba zfs/rpm/generic/zfs-dkms.spec --nodeps ...... + rpm2cpio /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el9.1/ib_stack/inkernel/BUILD/RPMS/x86_64/kmod-zfs-devel-2.1.5-1.el9.x86_64.rpm + cpio -id + CONFIGURE_FLAGS='--with-zfs-obj=...... --with-zfs=...... --with-spl-obj=...... --with-spl=...... --with-zfs-devel=...... ' + OSDZFSRPM=true + return 0 </pre> </div></div> <p>In order to debug <tt>lbuild</tt> <tt>build_spl_zfs()</tt>, I performed the commands step by step as follows on my local RHEL 9.1 vm node but found <b>kmod-zfs</b> rpm could be successfully built and installed:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># rpm -qa | grep zfs # mkdir /root/BUILD # cd BUILD/ # mkdir -p RPMS RPMS/x86_64 RPMS/noarch BUILD SOURCES SPECS SRPMS # cd # git clone https://github.com/zfsonlinux/zfs.git # cd zfs # git checkout -b lbuild zfs-2.1.5 # bash autogen.sh # ./configure --with-linux=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64 --with-linux-obj=/root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64 # make dist # cd ../ # ln -f zfs/zfs-2.1.5.tar.gz /root/BUILD/SOURCES # ln -f zfs/scripts/kmodtool /root/BUILD/SOURCES # rpmbuild -ba zfs/rpm/generic/zfs.spec --nodeps -v --define '_use_internal_dependency_generator 0' --define 'require_kdir /root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64' --define 'kver 5.14.0-162.22.2_lustre.el9.x86_64' --define 'kernels 5.14.0-162.22.2_lustre.el9.x86_64' --define '_tmppath /var/tmp' --define 'kernelbuildroot /' --define '_topdir /root/BUILD' # rpmbuild -ba zfs/rpm/generic/zfs-kmod.spec --nodeps -v --define '_use_internal_dependency_generator 0' --define 'require_kdir /root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64' --define 'kver 5.14.0-162.22.2_lustre.el9.x86_64' --define 'kernels 5.14.0-162.22.2_lustre.el9.x86_64' --define '_tmppath /var/tmp' --define 'kernelbuildroot /' --define '_topdir /root/BUILD' # rpmbuild -ba zfs/rpm/generic/zfs-dkms.spec --nodeps -v --define '_use_internal_dependency_generator 0' --define 'require_kdir /root/kernel/rpmbuild/BUILD/kernel-5.14.0-162.22.2.el9_1/linux-5.14.0-162.22.2_lustre.el9.x86_64' --define 'kver 5.14.0-162.22.2_lustre.el9.x86_64' --define 'kernels 5.14.0-162.22.2_lustre.el9.x86_64' --define '_tmppath /var/tmp' --define 'kernelbuildroot /' --define '_topdir /root/BUILD' # cd /root/BUILD/RPMS/x86_64/ # rpm -ivh zfs-2.1.5-1.el9.x86_64.rpm kmod-zfs-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm kmod-zfs-devel-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64.rpm libzfs5-2.1.5-1.el9.x86_64.rpm libzfs5-devel-2.1.5-1.el9.x86_64.rpm libzpool5-2.1.5-1.el9.x86_64.rpm libnvpair3-2.1.5-1.el9.x86_64.rpm libuutil3-2.1.5-1.el9.x86_64.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:libnvpair3-2.1.5-1.el9 ################################# [ 13%] 2:libuutil3-2.1.5-1.el9 ################################# [ 25%] 3:libzfs5-2.1.5-1.el9 ################################# [ 38%] 4:libzpool5-2.1.5-1.el9 ################################# [ 50%] 5:kmod-zfs-5.14.0-162.22.2_lustre.e################################# [ 63%] 6:zfs-2.1.5-1.el9 ################################# [ 75%] Created symlink /etc/systemd/system/zfs-import.target.wants/zfs-import-cache.service → /usr/lib/systemd/system/zfs-import-cache.service. Created symlink /etc/systemd/system/zfs.target.wants/zfs-mount.service → /usr/lib/systemd/system/zfs-mount.service. Created symlink /etc/systemd/system/zfs.target.wants/zfs-share.service → /usr/lib/systemd/system/zfs-share.service. Created symlink /etc/systemd/system/zed.service → /usr/lib/systemd/system/zfs-zed.service. Created symlink /etc/systemd/system/zfs.target.wants/zfs-zed.service → /usr/lib/systemd/system/zfs-zed.service. Created symlink /etc/systemd/system/multi-user.target.wants/zfs.target → /usr/lib/systemd/system/zfs.target. Created symlink /etc/systemd/system/zfs.target.wants/zfs-import.target → /usr/lib/systemd/system/zfs-import.target. Created symlink /etc/systemd/system/zfs-volumes.target.wants/zfs-volume-wait.service → /usr/lib/systemd/system/zfs-volume-wait.service. 7:libzfs5-devel-2.1.5-1.el9 ################################# [ 88%] 8:kmod-zfs-devel-5.14.0-162.22.2_lu################################# [100%] # rpm -qa | grep kmod-zfs kmod-zfs-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64 kmod-zfs-devel-5.14.0-162.22.2_lustre.el9.x86_64-2.1.5-1.el9.x86_64 </pre> </div></div> <p> </p> <p>The issue was narrowed down to Jenkins build system and/or ljb tool.<br/> Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mdiep" class="user-hover" rel="mdiep">mdiep</a> and <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=mkvardakov" class="user-hover" rel="mkvardakov">mkvardakov</a>, could you please investigate further? Thank you.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=colmstea" class="user-hover" rel="colmstea">colmstea</a>, could you check ljb tool?</p> <p>This does not look like an LJB issue. LJB is installing the kernel from the build, setting the proper default kernel and rebooting the VM to apply the kernel change (Jian's build from above no longer exists so I used <a href="https://build.whamcloud.com/view/Reviews/job/lustre-reviews/94201/" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/view/Reviews/job/lustre-reviews/94201/)</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm1 ~]# uname -a Linux onyx-22vm1.onyx.whamcloud.com 5.14.0-162.18.1_lustre.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 11 20:39:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux </pre> </div></div> <p>LJB installed the expected repo</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@onyx-22vm1 ~]# cat /etc/yum.repos.d/lustre-build.repo [lustre-build] name=lustre-build baseurl=http://mgmt/cobbler/localmirror/hudson/lustre-reviews/94201/arch-x86_64_build_type-server_distro-el9.1_ib_stack-inkernel enabled=1 priority=1 gpgcheck=0 </pre> </div></div> <p> </p> <p>When I install zfs, yum installs zfs-dkms as Jian saw. If I install kmod-zfs I get lots of 'nothing provides ksym' errors as Jian saw also.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>... - nothing provides ksym(__SCT__cond_resched) = 0xc07351b3 needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64   - nothing provides ksym(__SCT__might_resched) = 0xe2c17b5d needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64   - nothing provides ksym(__SCT__preempt_schedule) = 0x48d88a2c needed by kmod-zfs-5.14.0-162.18.1_lustre.el9.x86_64-2.1.5-1.el9.x86_64 ...</pre> </div></div> <p> </p> <p>I looked at the console log for the build of the el9.1 server and see a lot of messages like this. Perhaps this has something to do with it?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>11:47:46 ******************************************************************************** 11:47:46 *********************** KERNEL ABI COMPATIBILITY WARNING *********************** 11:47:46 ******************************************************************************** 11:47:46 The following kernel symbols are not guaranteed to remain compatible with 11:47:46 future kernel updates to this RHEL release: 11:47:46 11:47:46 __SCT__cond_resched 11:47:46 __SCT__preempt_schedule_notrace 11:47:46 ___ratelimit 11:47:46 __bdev_dax_supported </pre> </div></div> <p> </p> <p>"Minh Diep &lt;mdiep@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50746" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50746</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16657" title="installing kmod-lustre-osd-zfs failed for RHEL 9.1 server" class="issue-link" data-issue-key="LU-16657"><del>LU-16657</del></a> build: don't need to use zfs kmodtool on RHEL9.1<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 21bb46e6b7bb8d392fda8cec5aaa25304eed18f8</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50746/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50746/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16657" title="installing kmod-lustre-osd-zfs failed for RHEL 9.1 server" class="issue-link" data-issue-key="LU-16657"><del>LU-16657</del></a> build: don't need to use zfs kmodtool on RHEL9.1<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8a967f69e1f7f3e5b72766ae47db3bf64c76bac0</p> <p>Landed for 2.16</p> Related LU-16006 LU-16626 Development Rank 1|i03e47: Rank (Obsolete) 9223372036854775807 Severity