[LU-9731] kmods need to be limited to EL minor release kernel

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.10.0
• Fix Version/s: Lustre 2.10.1, Lustre 2.11.0
• Type: Bug
• Priority: Blocker
• Reporter: Brian Murrell (Inactive)
• Assignee: Brian Murrell (Inactive)
• Resolution: Fixed
• Severity: 3

Description

Now that kmods are being produced, they need to be limited to the kernel of the RHEL minor release they were created for.

This is because RHEL kernels have a kabi "whitelist".  That means, that only a subset of kernel interfaces are guaranteed to be stable by RHEL's kabi and it's only those interfaces that are put into a kernel's list of "kernel(...) = ..." Provides: and a kmod's Requires:.  This means that a kmod produced for RHEL 7.3 will look compatible with a RHEL 7.4 kernel (because the whitelisted kabi will not have changed across those releases) even though it is not because the Lustre kmods use interfaces that are not on the whitelist and can change from one minor release to another, even though the whitelisted kabi has not changed.

While Red Hat guarantees that these non-whitelisted interfaces will not change within a minor release (i.e. 7.3 to 7.4) there is no such guarantee across minor releases and in practice they probably almost always change across minor releases so a kmod using non-whitelisted interfaces needs to limit itself to the kernel provided in a RHEL minor release.

For a kmod produced on a RHEL 7.3 kernel that means adding a Requires: kernel >= 3.10.0-514, kernel <= 3.10.0-514 to the kmod RPM.

If this is not done, the kmod will install on to a RHEL 7.4 machine, which has an incompatible kernel by default and a compatible kernel (kernel-3.10.0-514*) will not be installed even if it's available in a Yum repo, even though it should be.

Issue Links

• is blocking: LU-9824 kernel update [SLES12 SP2 4.4.74-92.32]
• is related to: LU-11720 SRPM unnecessarily tied to kernel ver...

Comments

[Peter Jones]

Minh

Can you please advise on this one?

Thanks

Peter

[Andreas Dilger]

Brian, is there a ticket which lists the non-white listed symbols that we are using in Lustre? It would be useful to know this and see if we can stop using those symbols, or ask RH to add them to the whitelist?

[Brian Murrell (Inactive)]

If you install kernel-abi-whitelists before you build, it will tell you which interfaces you are using that are not on the whitelist such as:

********************************************************************************
*********************** KERNEL ABI COMPATIBILITY WARNING ***********************
********************************************************************************
The following kernel symbols are not guaranteed to remain compatible with
future kernel updates to this RHEL release:

	__fentry__
	__stack_chk_fail

Red Hat recommends that you consider using only official kernel ABI symbols
where possible. Requests for additions to the kernel ABI can be filed with
your partner or customer representative (component: driver-update-program).

So surprisingly only a couple.

It would probably be useful for us if once we have eliminated the use of those interfaces, that we fail the build if a new use of non-whitelisted interfaces is discovered.  I'm not sure if RH's kernel RPM build has such an option though.  I suppose this would be useful outside of RPM building too though, so that developers that introduce non-whitelisted interface use see the error when they run make.  I guess somebody would have to figure out how to utilise kernel-abi-whitelists for that.

[Brian Murrell (Inactive)]

I have tried an experiment where I have added:

Requires:       kernel < 3.10.0-515, kernel >= 3.10.0-514

 to the kmp-lustre.preamble in the lustre-client build and it has the desired effect.

Of course, not using non-whitelisted interfaces is the more desired approach because it would mean that our kmod would work across even the minor (i.e. 7.2->7.3->7.4->7.5, etc.) upgrades, but until that can happen, the above addition to kmp-lustre.preamble seems a suitable stop-gap.

[Andreas Dilger]

I don't see how/where these functions are being used? I searched for _ftrace and _stack_chk_fail and there are no direct users in our code. I looked at the kernel sources and couldn't even find anywhere that we indirectly used those functions. Is it possible that you can find out how we are using these functions?

[Brian Murrell (Inactive)]

Hrm.  I don't think I'm knowledgeable enough in this area to make quick work of this.  I doubt I'd find any use in our sources that you didn't find.  The only thing grep found me was a bunch of references such as:

static const struct modversion_info ____versions[]
__used
__attribute__((section("__versions"))) = {
...
	{ 0xf0fdf6cb, "__stack_chk_fail" },
...
};

but I'm sure you found those also and have much better understanding of what they are and what they do that I would have.

Perhaps mdiep can open a ticket with Red Hat to understand why their tool is declaring our module using interfaces that it's not.

Now after having said all of this, I have a recollection of running the weak-updates tool for the -514 kernel modules on the -681 (RHEL 7.4) kernel and seeing a lot more incompatibilities than just those two interfaces.

So perhaps more investigation is needed here one way (removing all use of non-whitelisted interfaces) or the other (adding a Requires: for the minor kernel version as I described above).

But without a solution one way or the other, users are left with a kmod RPM on their machine that has no kernel version information associated with it and so no clue as to which kernel to even install (never-mind boot) to get the kmod working.

[Brian Murrell (Inactive)]

1. how is this a blocker for Lustre?

Because without it, when one installs a/the kmod(s) for lustre on a system that has an incompatible kernel (i.e. RHEL 7.4 currently, but it could be 7.2 or 7.1, etc.), a compatible kernel is not installed and there is no indication one way or another which kernel the user needs to install to get the kmod(s) functional. So the user ends up installing kernels by trial and error until he finds one that the modules will load into.

For the human use-case, this is frustrating at best, but for the managed-installer (i.e. IML, but any configuration management system such as chef or puppet, or even just scripting that a user might create) case, it becomes quite impossible other than brute-force searching which is very time consuming.

2. I see you tested with 'kernel >= 3.10.0-514' for Requires.

kernel < 3.10.0-515, kernel >= 3.10.0-514 to be exact.

Would this cover the cases with number such as -514.21.2 that changes to -514.26.2?

If I understand what Red Hat is saying about their kABI, yes. As I understand it, their kABI guarantee says that whitelisted interfaces are stable across even (RHEL) minor upgrades so that if a kernel module only uses whitelisted interfaces, it will work on RHEL 7.1, 7.2, 7.3, 7.4, etc. This is why adilger's comment is so important and (IMHO) is very worthy of investigation as it would free us up even more from having to chase kernels even across RHEL minor version upgrades.

However, even in cases where non-whitelisted interfaces are being used, which is the case currently with the lustre-client (even though it does not appear to be – which is what needs investigating to achieve portability across RHEL minor upgrades), RHEL guarantee that within a minor release even non-whitelisted interfaces will not change. In kernel version nomenclature, that first digit after the kernel version (the -514 for the RHEL 7.3 kernel) is the demarcation of the non-whitelisted kABI interface guarantee and so any kernel built on a 3.10.0-514.* kernel should work on any other 3.10.0-514.* kernel.

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/27958
Subject: LU-9731 build: limit lustre kmod to EL minor release
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bc45b3ed70f34ea9d40bc9c4962cfe9dc183d6c5

[Brian Murrell (Inactive)]

adilger You might be interested in following the  Red Hat bug I opened about this.

In that ticket they have identified that at least two of the interfaces the client is using that are (I am presuming – pending confirmation) not on the kABI whitelist:

[ 2289.362766] libcfs: disagrees about version of symbol crypto_alloc_ahash
[ 2289.367132] libcfs: Unknown symbol crypto_alloc_ahash (err -22)
[ 2289.371425] libcfs: disagrees about version of symbol crypto_ahash_setkey
[ 2289.374198] libcfs: Unknown symbol crypto_ahash_setkey (err -22)

They do confirm that the hash for those two interfaces did change from RHEL 7.3 to RHEL 7.4 which, AFAIU, means they cannot be whitelisted interfaces.

[Brian Murrell (Inactive)]

mdiep: more from the Red Hat ticket referred to previously:

The %kernel_module_package macro generates the core of the subpackage, but it's not sufficient to properly package the kernel modules.

Usually we use the find-requires.ksyms script to generate the respective Requires: for the module package. This can be easily executed by overloading the ____find_requires macros, such as:

%define __find_requires /usr/lib/rpm/redhat/find-requires.ksyms

The find-requires.ksyms script is delivered by the 'redhat-rpm-config' package. That's what I'm missing in the lustre .spec file.

If I'm understanding that correctly, that solution would be preferred to the solution that I proposed of adding RHEL minor release kernel versions as a Requires:.

[Minh Diep]

isn't this what we already doing?

Processing files: kmod-lustre-client-2.10.50-1.el7.centos.x86_64
Executing(%doc): /bin/sh -e /tmp/rpmbuild-lustre-root-WQmEMIgm/TMP/rpm-tmp.OuFNGP
+ umask 022
+ cd /tmp/rpmbuild-lustre-root-WQmEMIgm/BUILD
+ cd lustre-2.10.50
+ DOCDIR=/tmp/rpmbuild-lustre-root-WQmEMIgm/BUILDROOT/lustre-2.10.50-1.x86_64/usr/share/doc/kmod-lustre-client-2.10.50
+ export DOCDIR
+ /usr/bin/mkdir -p /tmp/rpmbuild-lustre-root-WQmEMIgm/BUILDROOT/lustre-2.10.50-1.x86_64/usr/share/doc/kmod-lustre-client-2.10.50
+ cp -pr COPYING /tmp/rpmbuild-lustre-root-WQmEMIgm/BUILDROOT/lustre-2.10.50-1.x86_64/usr/share/doc/kmod-lustre-client-2.10.50
+ cp -pr ChangeLog-lustre /tmp/rpmbuild-lustre-root-WQmEMIgm/BUILDROOT/lustre-2.10.50-1.x86_64/usr/share/doc/kmod-lustre-client-2.10.50
+ cp -pr ChangeLog-lnet /tmp/rpmbuild-lustre-root-WQmEMIgm/BUILDROOT/lustre-2.10.50-1.x86_64/usr/share/doc/kmod-lustre-client-2.10.50
+ exit 0
Finding  Provides: /usr/lib/rpm/redhat/find-provides
Finding  Requires(interp):
Finding  Requires(rpmlib):
Finding  Requires(verify):
Finding  Requires(pre):
Finding  Requires(post):
Finding  Requires(preun):
Finding  Requires(postun):
Finding  Requires(pretrans):
Finding  Requires(posttrans):
Finding  Requires: /usr/lib/rpm/redhat/find-requires

********************************************************************************
*********************** KERNEL ABI COMPATIBILITY WARNING ***********************
********************************************************************************
The following kernel symbols are not guaranteed to remain compatible with
future kernel updates to this RHEL release:

        PDE_DATA
        __fentry__
        __stack_chk_fail
        remove_wait_queue
        seq_lseek
        seq_read

Red Hat recommends that you consider using only official kernel ABI symbols
where possible. Requests for additions to the kernel ABI can be filed with
your partner or customer representative (component: driver-update-program).

/usr/lib/rpm/redhat/find-requires calls /usr/lib/rpm/redhat/find-requires.ksyms

[root@onyx-28vm1 redhat]# tail -10 /usr/lib/rpm/redhat/find-requires
    then
        unset is_kmod;
        break;
    fi
done

[ -x /usr/lib/rpm/redhat/find-requires.ksyms ] && [ "$is_kmod" ] &&
    printf "%s\n" "${filelist[@]}" | /usr/lib/rpm/redhat/find-requires.ksyms

[Brian Murrell (Inactive)]

Are you sure all of the conditions for running /usr/lib/rpm/redhat/find-requires.ksyms at the end of /usr/lib/rpm/redhat/find-requires.ksyms are being met?  I.e. does /usr/lib/rpm/redhat/find-requires.ksyms exist and $is_kmod" evaluating to true?

[Minh Diep]

yes, I have injected an echo to print out $is_kmod; and verified is_kmod=1

[Brian Murrell (Inactive)]

And /usr/lib/rpm/redhat/find-requires.ksyms does exist of course, yes?  Just want to make double sure before I go back to Red Hat claiming it's not working as they advertise.

[Minh Diep]

yes it does

[Dmitry Eremin (Inactive)]

I don't think we need this restriction at all. This is redundant because of modules installs into directory with specific kernel version it's build for. So, if other version of kernel is loaded those modules will not be used. But when this packet installed on system the script "/sbin/weak-modules --add-modules" is used to propogate those modules into other versions of kernel which are installed on system. So, this script is responsible for checking compatibilty with other kernel versions (it create symlinks in compatible kernel's weak-updates/ directory). It's not used "Required:" field from the package. Therefore I see no reason to add this field in our package.

[Brian Murrell (Inactive)]

dmiterThe reason this is needed is because if you install the kmod-lustre-client RPM on a system where there is no (i.e. weak-updates) compatible kernel at all (i.e. on a RHEL 7.4 system) then you end up with a set of modules that have no matching kernel so no way to even boot to a kernel that will use them.

So, installing the kmod should result in a compatible kernel being present.  If that's the kernel that is already installed, then all is fine, but if there is no matching kernel already installed, the Yum transaction doing the kmod installation should install an appropriate kernel also.

The problem is that currently, the kmod-lustre-client RPM doesn't have enough information in it to make sure a compatible kernel is installed.  This may be a bug in the Red Hat kmod building tools as is being explored in a Red Hat ticket or it may not be.  That is still to be determined.  Even if it is a bug, it will likely be some time before we get a fix and we need to work-around this issue in the meanwhile.

Of course, the other (probably not at all short-term) resolution as has also been discussed in this ticket is to get the client kABI compatible.  But that is also not likely going to happen in the time-frame that this issue needs to be either resolved or worked-around in.

[Dmitry Eremin (Inactive)]

I don't see the issue with be able to install the package on any system even without appropriate kernel installed. The package will be installed but not used. Why this is an issue? As I mentoined before the script /sbin/weak-modules is responsible for propogation those modules into compatible kernels only. So, how this package will affect incompatioble kernel if those modules will not be loaded into this kernel at all?

[Brian Murrell (Inactive)]

Why would I install a kernel modules package that I didn't want to use?

If I install kmod-lustre-client surely I want to use the Lustre client on that node and thus I need a kernel that can use the modules.

[Dmitry Eremin (Inactive)]

The customer can install our kernel modules assume those are compatible with the latest kernel currently installed. We should not strict user with particular version of kernel we build for. This was main reason of introducing weak symbols support.

 

P.S. what account I can use to see the Red Hat ticket you mentoined?

 

[Brian Murrell (Inactive)]

dmiter: You need to use an @intel.com account on RH's Bugzilla in order to see bugs filed by Intel folks.

If you look at comment #23 in the ticket I have opened in their Bugzilla, you can see that the behaviour that I've described in this ticket is the actual expected behaviour they intend for kmods and that it's a bug in their kmod packaging scripts that not enough "Requires:" are added to a kmod so that a matching kernel can be found to be installed with the kmod.

[Gerrit Updater]

Brian J. Murrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28066
Subject: LU-9731 kmods need to be limited to EL minor release kernel
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: fdf2aca83eed194b218bdb1505f168873c88c8ad

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28066/
Subject: LU-9731 kmods need to be limited to EL minor release kernel
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 527b2cd8e593b52326519d13418daf34b6b53b0e

[Bob Glossman (Inactive)]

I think this change mangles the Required strings on SLES. For example in build of current master for sles12sp2 where the kernel version is 4.4.59-92.17 the Requires in the built lustre-client-kmp-default package has:
kernel >= 4.4.59-0
kernel < 4.4.59-1

while the kernel-default for the pristine unpatched upstream kernel has a Provides of:

kernel = 4.4.59-92.17

Don't see how these can properly match for the purposes of install dependencies.

[Brian Murrell (Inactive)]

Interesting that the sles12sp2 test run didn't fail to install the client RPMs because of this.

Given that this patch is specifically to work around a bug in RHEL's kmod building macro, it should probably be limited to RHEL kmod building only.  I'll push a patch for that.

[Gerrit Updater]

Brian J. Murrell ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28202
Subject: LU-9731 Limit work-around to EL7 only
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 83952af84f928fb0289744c61fbe215e56bf32d6

[Bob Glossman (Inactive)]

Interesting that the sles12sp2 test run didn't fail to install the client RPMs because of this.

Unless called for in Test-Parameters I don't think any SLES test runs are done routinely. For sure not in review tests.

While I think this additional mod will fix the problem I wonder if it might be more well structured to push the lines putting in extra Requires into a function in lbuild-rhel and then in lbuild call that function if it exists. That way the extra Requires would be in all RHEL builds, not just RHEL7 and would still be left out of non-RHEL builds.

[Brian Murrell (Inactive)]

That way the extra Requires would be in all RHEL builds, not just RHEL7

This problem has only been confirmed on EL 7 (so far) so applying it to EL 6 might be inappropriate.

Fencing the fix off into the distro-specific lbuild file might not be horrible though.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28202/
Subject: LU-9731 Limit work-around to EL7 only
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 950c511276866efb6af8defe49213fd69b8883d2

[Peter Jones]

Second patch landed for 2.11

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28438
Subject: LU-9731 Limit work-around to EL7 only
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 0c82f83e5ec50507c0ebc586e968abdb7f352c93

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/28438/
Subject: LU-9731 Limit work-around to EL7 only
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 9ea9bc9f870ede585166d32442ba9bd22f653c32