|
Minh
Could you please review these patches when Steve supplies them?
Thanks
Peter
|
|
Don't hold your breathe waiting for them ;^)
I started to take a look, and realized I need to have a handle on what lbuild does before I work this out.
|
|
Can you just paste your complete lustre.spec here and we can see which side gets to it first? 
|
|
Simplified version of the SGI spec
|
|
Anyone is very welcome to jump ahead of me.
I just attached a stripped down version of the spec used for our 2.4.2 release. Essentially, the %kernel_module_package macro replaces the module package and all of the associated pre/post scripts. The tricky bit is that it expects to build for every installed kernel. You either have to override this or deal with it.
There are also some hazards in the macro itself : ie, it expects that all installed kernels have the same version.
All of this gets especially nasty dealing with the server packages, which is the default flavor with the version string twiddled (I dealt with this in SGI releases by building the server kernel as a distinct flavor). I suspect the answer is simple to not do weak-modules for servers, as there is limited benefit to it on servers.
I was considering having a separate spec: either for clients vs servers, or for --with-weak-updates vs --without-weak-updates.
|
|
http://review.whamcloud.com/12063
Intended only as a conversation piece at this point.
|
|
^Build failed because there are multiple versions of the same kernel installed.
%kernel_module_package normally uses rpm to query the installed kernel version, and packages modules for the first listed kernel version. That resulted in a mismatch between the kernel the modules were built for and the kernel the modules were (failed to be) packaged for.
I'm exploring some ways around this.
|
|
With RedHat, we can define kernel_version to override the default selection of which kernel version to build for.
SuSE does not have anything like that, but it looks there is a mechanism with %suse_kernel_module_package (which is the underlying implementaion of %kernel_module_package on SLES). I will investigate that, and perhaps engage SuSE to get the SLES %kernel_module_package to use that facility in a way which is compatible with RH's.
In the meantime, I'm looking at the build failure from http://review.whamcloud.com/#/c/12063/2, which I could not reproduce. On my system, /usr/lib64/lustre/mount_osd_xyz.so is packaged with the osd xyz kmp. I suspect something odd about %
{with lustre_utils}
.
|
|
Any further news on this one Stephen?
Thanks,
~ jfc.
|
|
Building on local systems with './configure ; make rpms', the patch I have is able to build for redhat when multiple kernel package versions are installed. It works on SLES with a single version installed.
Building with Intel's jenkin's, these files:
%{_libdir}/lustre/mount_osd_ldiskfs.so
%{_libdir}/lustre/mount_osd_zfs.so
which should be part of the their respective osd module packages, are not picked up, and I was unable to identify why this difference arises or how to address in it the time I had available.
|
|
Alexander Boyko (alexander.boyko@seagate.com) uploaded a new patch: http://review.whamcloud.com/14191
Subject: LU-5614 build: add kmod for client
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 6748dbb146de7d8a8df72765996180a1a9b2837b
|
|
I found that it needs to have kernel-devel installed on the builder. As we use lbuild to extract the kernel-devel on to the kdir, it doesn't seem to work
++ /usr/bin/rpmbuild --target x86_64 -tb /mnt/build/lustre-release/lustre-2.7.55.tar.gz --without servers --define '__find_requires /mnt/build/lustre-release/BUILD/find-requires' --define 'configure_args --with-o2ib=no' --define 'kdir /mnt/build/lustre-release/BUILD/reused/usr/src/kernels/2.6.32-504.23.4.el6.x86_64' --define '_tmppath /var/tmp' --define '_topdir /mnt/build/lustre-release/BUILD'
error: Failed build dependencies:
kernel-devel is needed by lustre-client-2.7.55-2.6.32_504.23.4.el6.x86_64_ge550f31.x86_64
|
|
lbuild and the environment it operates in is sufficiently nonstandard for host and rpm builds that we'll have difficulty getting it to use any of the standard kernel module package methods without addressing some of it's deficiencies.
multiple kernel-devel packages can be installed concurrently, so there is no reason not to install the package if it is required by a build - and it is correctly required by any package building kernel modules.
That said... The Buildreq is automatically added by %kernel_module_package. It looks like it can be overridden with nobuildreqs=yes.
|
|
even with your latest patch, it won't install without building with kernel-devel
[root@onyx-24 x86_64]# rpm -hiv kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64.rpm
error: Failed dependencies:
ksym(__init_waitqueue_head) = 0xffc7c184 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(__mutex_init) = 0x4bf79039 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(__stack_chk_fail) = 0xf0fdf6cb is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(__wake_up) = 0x642e54ac is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(add_wait_queue) = 0x650fb346 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(copy_from_user) = 0x3302b500 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(default_wake_function) = 0xffd5a395 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(kfree) = 0x037a0cba is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
|
I found that it needs to have kernel-devel installed on the builder. As we use lbuild to extract the kernel-devel on to the kdir, it doesn't seem to work
++ /usr/bin/rpmbuild --target x86_64 -tb /mnt/build/lustre-release/lustre-2.7.55.tar.gz --without servers --define '__find_requires /mnt/build/lustre-release/BUILD/find-requires' --define 'configure_args --with-o2ib=no' --define 'kdir /mnt/build/lustre-release/BUILD/reused/usr/src/kernels/2.6.32-504.23.4.el6.x86_64' --define '_tmppath /var/tmp' --define '_topdir /mnt/build/lustre-release/BUILD'
error: Failed build dependencies:
kernel-devel is needed by lustre-client-2.7.55-2.6.32_504.23.4.el6.x86_64_ge550f31.x86_64
I think you can use --nodeps with rpmbuild. This is of course something that needs to be done with care and attention to not paper over other missing dependency requirements. But those will float up to the surface pretty quickly.
even with your latest patch, it won't install without building with kernel-devel
[root@onyx-24 x86_64]# rpm -hiv kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64.rpm
error: Failed dependencies:
ksym(__init_waitqueue_head) = 0xffc7c184 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(__mutex_init) = 0x4bf79039 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(__stack_chk_fail) = 0xf0fdf6cb is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(__wake_up) = 0x642e54ac is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(add_wait_queue) = 0x650fb346 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(copy_from_user) = 0x3302b500 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(default_wake_function) = 0xffd5a395 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
ksym(kfree) = 0x037a0cba is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64
These are not missing kernel-devel dependencies. These are missing "weak-module" ABI dependencies.
Which kernel(s) were on the machine you tried to install kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64.rpm on? The above errors are telling you that you don't have a kernel that is ABI (weak module) compatible with your kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64.rpm
|
|
Minh or Steve,
Just checking in to see where we are going with this ticket?
Thanks,
~ jfc.
|
|
Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: http://review.whamcloud.com/17489
Subject: LU-5614 build: use %kernel_module_package in rpm spec
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: e36f89001ffc45710d8fd7dea44c944989018ef1
|
|
I used a different Change-Id for my revision of Stephen's patch, and marked mine "fortestonly" to reduce confusion somewhat. Nothing too significantly different in mine yet, but I plan to continue work on mine next week. If folks like the changes and don't mind, I can update Stephen's original patch (if someone doesn't make the needed improvements before me).
Stephen's work is an important step towards having Lustre sanely packed for RHEL. LLNL is moving to the koji build farm for our RHEL7-based TOSS3 distribution, and having this patch that gets weak-updates working is a high priority for us.
|
|
Once we have this weak-modules support, we can finally drop the kernel version string from Lustre's release string. That will enable even more cleanup. Yay!
|
|
I'm glad you are finding this work useful!
You need to be careful about the build environment for the kmp packaging to work as expected. You can hack around limitations a bit, but really need to have only a single version-release installed. This was never an issue for my build system, but the Jenkins environment does not enforce it.
To deal with some of the problems of existing installations, crossbuilds, etc, I was looking at having multiple rpm specs, selected by configure options.
|
|
Christopher, I can't see my responses to your inline comments, so...
You can do Obsoletes, Requires, etc with %kernel_module_package (%kmp) by providing a preamble. This a file which is expanded into the subpackage preamble. My use was:
%global kernel_version %kversion
%kernel_module_package must want this variable set, but a comment to explain that would help avoid confusion
On RH based distros, it is used to override the default method of identifying which kernel to build for. It is not needed if the build environment is strictly controlled. See my Oct 1 comment.
Is there some reason not to simply use this line unconditionally?
You would need to move the without_servers condition into kmp-lustre.files
That is probably a better way. As is, I think the client kmp is missing the %doc files
|
Christopher, I can't see my responses to your inline comments, so...
You have to click the "review" button to post your inline comments.
On RH based distros, it is used to override the default method of identifying which kernel to build for. It is not needed if the build environment is strictly controlled. See my Oct 1 comment.
Yes, I understand. I think you are right to include that for now, because changing the way Lustre selects which kernel to build against would only complicate the patch further. I am only asking that a comment be added to the spec file so that it is clearer to everyone in the future. You can see I added a comment in my version of the patch.
To deal with some of the problems of existing installations, crossbuilds, etc, I was looking at having multiple rpm specs, selected by configure options.
I completely agree. Having one spec file seems like a nice idea at first, but once you get into all of the little details, it becomes clear that rpm just was not designed to support multiple distros (or really even multiple versions of the same distro) in one spec file. We really should switch to having many simpler spec files instead of one inscrutable and always broken spec file.
I am not convinced yet whether configure should be involved. I would like to move towards having the build (compilation) system and the packaging system be more cleanly separated. But it is a possibility.
|
|
Server packages build fine under CentOS7 with my version of the patch, but not in Intel's buildfarm. Sigh. Can we kill lbuild yet?
|
|
I'd like to go back and answer John's question:
Just checking in to see where we are going with this ticket?
and I'll start by quoting Stephen Champion:
lbuild and the environment it operates in is sufficiently nonstandard for host and rpm builds that we'll have difficulty getting it to use any of the standard kernel module package methods without addressing some of it's deficiencies.
Stephen really hit the nail on the head. The nonstandard way that lbuild operates is the biggest obstacle to success in this ticket. It looks to me like Stephen's last revision of the patch was very close to complete. The remaining minor issues shouldn't take very long to work through, and I have already addressed some of them in my revision.
Build systems are surprisingly complicated and difficult to get right for every revision of every distribution. Adding that inherent complication to the unnecessary complication of lbuild's non-standard approach leaves us with a problem that is practically insurmountable.
I've been recommending that we ditch lbuild for something that uses more standard packaging approach for years now (LU-3956, but I'd been advocating to change it long before I opened that ticket). I have not seen much effort in that area, unfortunately.
So, Intel, how do we make progress here?
I would really like to see this and other build and packaging improvements land early in the 2.9 landing window. Is there some way to address the problems in Intel's build farm in that time frame? I would assume that setting up a full lbuild replacement in just a month or two won't happen, but we'll need some attention to improve the current situation enough to let sane changes happen.
So it would be good to lay out a plan for both short and longer term changes to Intel's buildfarm situation.
|
|
Hi Chris
Server packages build fine under CentOS7 with my version of the patch, but not in Intel's buildfarm. Sigh. Can we kill lbuild yet?
I see that it passed EL6.7 but failed in EL7. Have you tried local build on EL7? If so please share the output log.
I wonder what lbuild does in this case that worked of EL6.7 and didn't work on EL7.
Thanks
-Minh
|
|
CentOS7 is the same as EL7 for our purposes. So yes, I did try it, and it works.
I'm not sure which log you mean by "output log". But I'm also not seeing how that would help you since your build farm with lbuild is so incredibly different a normal build environment.
Welcome to the club. We all wonder why lbuild works on one but not the other. In an lbuild-free environment, the builds appear to work on that platform.
|
|
Regarding
>> lbuild and the environment it operates in is sufficiently nonstandard for host and rpm builds that we'll have difficulty getting it to use any of the standard kernel module package methods without addressing some of it's deficiencies.
We are using lbuild just as if anyone use rpmbuild to build the package. I believe we address one of the issue where kernel module package requires kernel-devel to install on the builders which we did.
it would be great if we can understand more of the deficiencies that you mentioned.
|
|
>>> I'm not sure which log you mean by "output log". But I'm also not seeing how that would help you since your build farm with lbuild is so incredibly different a normal build environment.
It would be nice if you can provide the commands and its' output that you built from EL7. I assume you'll run something like: autogen.sh, configure ...., make....
|
|
Yes, I ran "autogen.sh && ./configure && make rpms".
You have both successful and failed logs in lbuild paths for RHEL6.7 and RHEL7. Those are the logs and systems you need to be looking at in the short term.
|
|
Yes, I am looking. I am mostly interested in rpmbuild output. BTW did you build locally with patched kernel?
|
|
The build systems I am familiar with have a few key characteristics we are lacking for using this macro.
- Automatically installs packages to satisfy Requires: and BuildRequires: statements prior to invoking rpmbuild (or other native build mechanism).
- Builds only use libraries and objects which are installed in the build environment (host/guest/chroot) with the native package manager (rpm).
Not all of the build systems do this, but ideally kernel module package builds also have:
- Insure that only a single instance of a package name.arch is installed. This is normally enforced for packages other than the kernel and related packages - but building kernel modules, it is important to be strict about kernel packages because of hazards in how the target kernel is determined, how kernel module dependencies are resolved, and how kernel source and header links are managed on some distributions.
|
|
I moved forward a patch for kmod.
http://review.whamcloud.com/#/c/12063/16 was build success for all kernels but there was .files generation from a spec file and rpms for rhel have ksym() instead of kernel(). The .files generation from spec was changed to rpm/*.files with lbuild modifying as suggested by Christopher, patchset 22 was fine except one build server RH7. At patchset 23 I`ve added Buildrequires: kernel-devel = %
{kversion}
and all redhat build failed as expected.
To move forward this issue requires help from Intel Jenkins people, we need kernel-devel package with precise version to build valid kmod dependencies from kernel symbols.
|
|
Peter Jones, could we get Intel to weigh in here? We are almost certainly going to need to get more attention from a person or persons at Intel to make any further progress.
Alexander and Stephen have done some great work on this patch. The community has put in its effort to move Lustre leaps and bounds closer to proper packaging. Now we are all blocked by problems in Intel's build farm.
We need to have a serious discussion about application of manpower and approach to resolving this problem. We need to make some firm plans now if we are going to get this done for 2.9.
How do we move forward? Would a teleconference help?
|
|
Hi Chris,
I will starting looking into this and will gather more manpower and resources to figure out what issue with Intel's build farm and come up with a sensible solution.
|
|
Great, Minh!
|
|
started this by manually configure, make rpms but failed at make rpms
make[1]: Entering directory `/mnt/lustre-release'
make[1]: Leaving directory `/mnt/lustre-release'
Installing lustre-2.7.64-1_g481176c.src.rpm
error: Failed build dependencies:
kernel-devel = 2.6.32-573.12.1.el6.x86_64 is needed by lustre-client-2.7.64-2.6.32_573.12.1.el6.x86_64_g481176c.x86_64
make: *** [rpms] Error 1
[root@onyx-24 lustre-release]# rpm -qa | grep kernel-devel
kernel-devel-2.6.32-573.12.1.el6.x86_64
[root@onyx-24 lustre-release]#
This is similar to what Intel jenkins reported. still looking, just FYI
|
|
Minh, either use patch revision 22 for now or remove the "= %{kversion}" from the following line in the lustre.spec.in in patch revision 23:
|
|
ah, I see, revision 22 still failed on el7 due to the extra x86_64 string in the patch name
|
|
Minh, while that is a problem that needs resolution, the bigger issue that we need you focussed on is the kernel-devel package installation issue. For instance, while the Intel build farm says that the el6.7, inkernel, server build was successfully for patch revision 22, that isn't really correct. The build farm only checks that something came out of the build; it doesn't really check if what came out is correct.
The kernel symbol requirements in the resulting binary rpms are incorrect because the correct kernel-devel package was not installed at packaging time.
|
|
Chris, kernel-devel is installed whenever a new version released. I checked the builder node which built version 22 (onyx-8-sdf1-el6-x8664) and see kernel-devel installed
[root@onyx-8-sdf1-el6-x8664 ~]# rpm -qa | grep kernel-devel
kernel-devel-2.6.32-573.12.1.el6.x86_64
kernel-devel-2.6.32-573.3.1.el6.x86_64
kernel-devel-2.6.32-573.7.1.el6.x86_64
kernel-devel-2.6.32-573.8.1.el6.x86_64
kernel-devel-2.6.32-504.12.2.el6.x86_64
|
|
The next question that you need to ask yourself is this: "Are these the the kernel-devel packages for the correct kernel"?
Keep in mind that Lustre is being built against a custom patched kernel. This kernel is built and packaged by lbuild during the lustre build process. You can see the packages for one distro here (this is linked off of the same patch revision 22, el6.7, inkernel, server build that I referenced ealier):
https://build.hpdd.intel.com/job/lustre-reviews/36780/arch=x86_64,build_type=server,distro=el6.7,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/
And here are the relevant rpms:
You will note that the kernels that lbuild patches and builds have "_lustre.g773a7e4" appended to their revision number. The kernel packages that you listed do not contain that string, so I would expect that none of the kernel-devel packages that you listed are from the correct kernel.
|
|
gotcha!
|
|
Minh Diep, to check kmod dependencies you need to do
rpm -qp --requires kmod-xxx.rpm | grep vfree
the result should be kernel(vfree)
for example
Patchset 22 have ksym(vfree) and it is wrong.
|
|
Just a note: client builds are done using the distro kernel. So we need the distro -devel kernel packages for client builds and the _lustre.tag kernel packages for server builds.
|
|
I noticed that spl and zfs also require kernel-devel*lustre*. are they not having the same problem?
|
|
I`ve tried to check kmod-zfs, but fail. Download requires authorization. What is reason for separate access to lustre packages and kmod-zfs?
|
|
Limitations imposed (or at least perceived to be imposed) by the CDDL vs GPL licensing incompatibility.
|
|
Hi,
With my latest revision built under https://build.hpdd.intel.com/job/lustre-reviews/37084/arch=x86_64,build_type=server,distro=el6.7,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/
the kmod dependencies on el6.7 seems to be correct
[root@onyx-24 ~]# rpm -qp --requires kmod-lustre-2.7.64-2.6.32_573.8.1.el6_lustre.g160868c.x86_64_g160868c.x86_64.rpm | grep vfree
kernel(vfree) = 0x999e8297
[root@onyx-24 ~]#
I still need to figure the failure on el7 though
|
|
Minh, that is fantastic! Did you do any spot checking to see if the packages work?
|
|
Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: http://review.whamcloud.com/18170
Subject: LU-5614 build: use %kernel_module_package in rpm spec
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 2ec18b2635b101e3d2661d49949ca68af170a1fc
|
|
Chris, not yet. There will be more testing on this I am sure
|
|
Any theories on that final issue with the RHEL7 build?
|
|
We used to build el7 server on a shorter build directory (ie /var/lib/jenkins/tmp/el7_top_dir) because somehow rpmbuild doesn't work on long dir (ie /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el7/ib_stack/inkernel/BUILD/BUILD). I don't know why it failed with this kmod patch but not other. I changed it back to long dir to build in the same directory as git checkout; and this solved the issue.
|
|
Installation test found some issues
+ yum install -y kmod-lustre-osd-ldiskfs
Loaded plugins: fastestmirror, security
Setting up Install Process
Determining fastest mirrors
- base: mirror.nexcess.net
- extras: mirror.compevo.com
- updates: mirror.chpc.utah.edu
Resolving Dependencies
--> Running transaction check
---> Package kmod-lustre-osd-ldiskfs.x86_64 0:2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
kmod-lustre-osd-ldiskfs x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116
lustre-build 2.8 M
Transaction Summary
================================================================================
Install 1 Package(s)
Total download size: 2.8 M
Installed size: 16 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
^M Installing : kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre. 1/1
find: `/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre-osd-ldiskfs': No such file or directory
^M Verifying : kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre. 1/1
and interestingly, install lustre require to install lustre-dkms.
Installing:
lustre x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116 lustre-build 619 k
Installing for dependencies:
dkms noarch 2.2.0.3-30.git.7c3e7c5.el6 addon-epel6-x86_64 77 k
libnvpair1 x86_64 0.6.4.2-1.el6 lustre-build 27 k
libuutil1 x86_64 0.6.4.2-1.el6 lustre-build 32 k
libzfs2 x86_64 0.6.4.2-1.el6 lustre-build 112 k
libzpool2 x86_64 0.6.4.2-1.el6 lustre-build 389 k
lustre-dkms noarch 2.7.65-1.el6 lustre-build 12 M
lustre-osd-ldiskfs-mount x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116 lustre-build 13 k
lustre-osd-zfs-mount x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116 lustre-build 7.4 k
net-snmp-libs x86_64 1:5.5-54.el6_7.1 updates-centos6.7-x86_64 1.5 M
spl-dkms noarch 0.6.4.2-1.el6 lustre-build 446 k
zfs-dkms noarch 0.6.4.2-1.el6 lustre-build 1.9 M
Transaction Summary
======================================================================================================================================================================================
Install 12 Package(s)
|
|
Just wanted to say I'm strongly supporting this feature. I very want something similar to Chris' patch being landed! I also think this is the good way to go. At CEA we are building our own kernel, (M)OFED and Lustre RPMS, in standard way, using standard tools. I would love building lustre with such patch.
There is no reason this could not be possible in lbuild.
|
|
Minh, can you please elaborate on the "interestingly, install lustre require to install lustre-dkms" statement? Are you saying that the "yum install -y kmod-lustre-osd-ldiskfs" command was the one that pulled in the lustre-dkms package?
|
|
sorry for not being clear, it's yum -y lustre command that pulled in the lustre-dkms package
|
|
so lustre requires lustre-osd
[root@onyx-21vm4 ~]# rpm -qp --requires ./lustre-2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116.x86_64.rpm | head
rpmlib(VersionedDependencies) <= 3.0.3-1
kmod-lustre = 2.7.65
lustre-osd
lustre-osd-mount
which lustre-dkms provides
[root@onyx-21vm4 ~]# rpm -qp --provides ./lustre-dkms-2.7.65-1.el6.noarch.rpm
lustre-kmod = 2.7.65
lustre-modules = 2.7.65
lustre-osd
lustre-osd-zfs = 2.7.65
lustre-dkms = 2.7.65-1.el6
[root@onyx-21vm4 ~]#
|
|
Let's take stock of where we stand.
Obviously, we need a rebase of the patch.
Next we have the interaction between the dkms packages and the non-dkms packages. I think the solution there is probably pretty straight forward: don't put both packages in the same yum repository.
What else do we need to work through?
|
|
I have updated the patch with more dependencies, please review. This resolved the conflict with dkms package.
We will need to automate this for testing. this is is progress now.
|
|
Could you explain, why the next line was added?
kmod adds requires on the symbols, so old style package references looks strange.
|
|
you're right, we don't need that.
|
|
I found that osd_ldiskfs.ko is not in weak-updates/... directory
[root@onyx-21vm4 ~]# uname -a
Linux onyx-21vm4.onyx.hpdd.intel.com 2.6.32-573.18.1.el6.x86_64 #1 SMP Tue Feb 9 22:46:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@onyx-21vm4 ~]# ls /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/*ldiskfs.ko
/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/ldiskfs.ko
/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko
[root@onyx-21vm4 ~]# ls /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/lustre/fs/*ldiskfs.ko
/lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/lustre/fs/ldiskfs.ko
[root@onyx-21vm4 ~]#
Is this expected or this is not working on lustre patched server?
|
|
There are some differences for a rpms between SLES and RHEL at the rpm kmod scripts.
RHEL
SLES
The main point is RHEL uses modules directory equal to rpm package name to find and add *ko, for example extra/lustre-osd-ldiskfs . And SLES uses rpm package to find and add modlues. We have a different package name lustre-osd-ldiskfs lustre-tests etc., but use one directory to install all of them extra/lustre/fs/ and this could bring RHEL to some kind of a problems.
We installed rpms and updated a kernel for a client. The Lustre works fine after update. I suppose that kmod-lustre do postintall weak -add for all *ko from extra/lustre/ and we didn`t have a problems that I`ve mentioned above.
Now with your question,
Did you see any error when you updated a kernel, it may have another version of kernel symbol and weak-update should fail, especially for ldiskfs*.
|
|
FYI, after expanding %kernel_module_package, the correct module installation directory can be obtained via %kernel_module_package_moddir instead of the inline selection and assignment to %kmoddir introduced with LU-4124.
As Alexander mentioned, I suspect a kernel symbol mismatch for osd_ldiskfs trying to install for a non-lustre kernel.
|
|
Did you see any error when you updated a kernel, it may have another version of kernel symbol and weak-update should fail, especially for ldiskfs*.
>> No, I did not. However, we can't mount lustre dev because of missing osd-ldiskfs module (after upgrade)
|
|
perhaps I did, here is the command
yum install -y kernel-2.6.32-573.12.1.el6_lustre.gb872116.x86_64 kmod-lustre kmod-lustre-osd-ldiskfs
Loaded plugins: fastestmirror, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:2.6.32-573.12.1.el6_lustre.gb872116 will be installed
---> Package kmod-lustre.x86_64 0:2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b will be installed
---> Package kmod-lustre-osd-ldiskfs.x86_64 0:2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b will be installed
--> Processing Dependency: lustre-osd-ldiskfs-mount = 2.7.65 for package: kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b.x86_64
--> Running transaction check
---> Package lustre-osd-ldiskfs-mount.x86_64 0:2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
kernel x86_64 2.6.32-573.12.1.el6_lustre.gb872116
lustre-build 30 M
kmod-lustre x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b
lustre-build 26 M
kmod-lustre-osd-ldiskfs x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b
lustre-build 2.8 M
Installing for dependencies:
lustre-osd-ldiskfs-mount x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b
lustre-build 13 k
Transaction Summary
================================================================================
Install 4 Package(s)
Total download size: 59 M
Installed size: 298 M
Downloading Packages:
--------------------------------------------------------------------------------
Total 55 MB/s | 59 MB 00:01
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
^M Installing : kernel-2.6.32-573.12.1.el6_lustre.gb872116.x86_64 1/4
^M Installing : lustre-osd-ldiskfs-mount-2.7.65-2.6.32_573.12.1.el6_lustre 2/4
^M Installing : kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre. 3/4
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_unregister_type
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_manual_cleanup
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_kbytestotal_seq_show
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_buf_realloc
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lustre_lma_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol seq_client_fini
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_register_type
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_fail_loc
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_oh_tally_log2
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_env_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lbug_with_loc
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_restore_sigs
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_get
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_log_goto
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_debug_msg
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol __cfs_fail_check_set
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_uuid_unparse
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_filesfree_seq_show
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_remove
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_register_stats
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_counter_add
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_device_put
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_conn2export
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol seq_server_alloc_meta
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol linkea_entry_unpack
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_purge
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol fld_local_lookup
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol server_name2index
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_env_fini
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_buf_free
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_op_begin
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol tgt_server_data_update
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_clear_sigpending
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol obd_memory
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_start
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_process_proc_param
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_dev_del_linkage
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lustre_lma_swab
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_register
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_hash_is_empty
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_txn_hook_start
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_debug_dumpstack
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_oh_sum
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol eq_client_alloc_fid
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_connect
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_fini
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_log_return
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_block_sigsinv
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_alloc_stats
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_txn_hook_stop
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_cdebug_printer
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_filestotal_seq_show
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_cdebug_printer
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_filestotal_seq_show
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_degister_many
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_otable_features
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_device_get
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_object_fini
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_device_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_object_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_register_many
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol statfs_pack
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_fini
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_prepare
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_init_finish
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_kbytesfree_seq_show
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_free_stats
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_revive_many
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_enter
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_kbytesavail_seq_show
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_single_release
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_counter_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_env_refill
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol linkea_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_oh_tally
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_object_put
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol pop_ctxt
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_kmem_fini
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_quota_glb_features
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_version_get
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol LU_DOT_LUSTRE_FID
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_acct_features
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_write_u64_helper
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_print
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_locate_at
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_directory_features
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_kmem_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_subsystem_debug
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol seq_client_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_search_type
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_init
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_write_helper
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_seq_create
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_lookup_dir
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_counter_sub
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_exit
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_debug
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_counter_sub
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_exit
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_debug
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_oh_clear
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_device_fini
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_op_end
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_fini
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_quiesce_many
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_blksize_seq_show
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_op_adjust
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol push_ctxt
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_txn_hook_commit
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_fail_val
WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_disconnect
find: `/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre-osd-ldiskfs': No such file or directory
^M Installing : kmod-lustre-2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86 4/4
^M Verifying : kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre. 1/4
^M Verifying : kernel-2.6.32-573.12.1.el6_lustre.gb872116.x86_64 2/4
^M Verifying : kmod-lustre-2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86 3/4
^M Verifying : lustre-osd-ldiskfs-mount-2.7.65-2.6.32_573.12.1.el6_lustre 4/4
|
|
Since you are building the modules with a patched kernel, it is not surprising that it is unable to resolve symbols when it attempts weak-updates for the unpatched kernel. Anytime you change the KABI without changing the flavor, this is the result. What is surprising is that only osd_ldiskfs.ko has trouble - this demonstrates great progress removing dependencies on the patched kernel!
To test weak updates on servers, try installing the server modules on a system with a different (and later) version #. Ie, build modules for 2.6.32-573.12.1.el6_lustre.foo and install them on 2.6.32-573.18.1.el6_lustre.fop
This is curious:
find: `/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre-osd-ldiskfs': No such file or directory
|
|
Ah, I was hoping that this ticket let us use weak updates on unpatched kernel. I will try your suggestion.
the line above is coming from the post install script
rpm --scripts -qp kmod-lustre-osd-ldiskfs-2.8.50-2.6.32_573.12.1.el6_lustre.g33b2752.x86_64_gd0671d9.x86_64.rpm
postinstall scriptlet (using /bin/sh):
if [ -e "/boot/System.map-2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" ]; then
/sbin/depmod -aeF "/boot/System.map-2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" "2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" > /dev/null || :
fi
modules=( $(find /lib/modules/2.6.32-573.12.1.el6_lustre.g33b2752.x86_64/extra/lustre-osd-ldiskfs | grep '\.ko$') )
if [ -x "/sbin/weak-modules" ]; then
printf '%s\n' "${modules[@]}" | /sbin/weak-modules --add-modules
fi
the path should be /lib/modules/2.6.32-573.12.1.el6_lustre.g33b2752.x86_64/extra/lustre/fs
|
|
The patch does allow using weak modules with unpatched kernels. This patch doesn't care whether the kernel is patched or not; it works with both. However not all kernels offer identical symbols, so one build of lustre will not magically work with all kernels.
When LU-684 is done, I believe that we will at last have eliminated the kernel symbol changes that Lustre is currently introducing in the Intel build farm.
|
|
Minh, are you working on splitting apart the yum repos for the dkms and standard lustre builds?
What else is currently on our list of blockers to getting this ticket done? Anything we can help with?
|
|
I am working on moving the inline Require in to a file. will upload a version shortly. I have verified that we do not need the split the repo.
This is one perhaps minor issue notice above about this error
find: `/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre-osd-ldiskfs': No such file or directory
when installing kmod-lustre-osd-ldiskfs. If you can help with this, this would be great.
|
|
and when I installed: yum install kmod-lustre-osd-zfs
I see error like these
--> Processing Dependency: ksym(arc_add_prune_callback) = 0x5bd5668d for package: kmod-lustre-osd-zfs-2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852.x86_64
--> Processing Dependency: ksym(__cv_broadcast) = 0x1d5ed383 for package: kmod-lustre-osd-zfs-2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852.x86_64
---> Package lustre-osd-zfs-mount.x86_64 0:2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852 will be installed
--> Finished Dependency Resolution
Error: Package: kmod-lustre-osd-zfs-2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852.x86_64 (lustre-build)
Requires: ksym(zap_remove) = 0xce8deead
Error: Package: kmod-lustre-osd-zfs-2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852.x86_64 (lustre-build)
Requires: ksym(sa_buf_rele) = 0xfb208743
|
|
Is ZFS installed?
|
|
yes, I installed zfs before that and verified that zfs, spl, kmod-zfs, kmod-spl were on the system
|
|
You are hacking some internal script to get it to make the kernel's symbol dependencies be named "ksym" instead of "kernel", right? Did that hack go too far and change the name for symbols that are in external modules as well?
|
|
I don't think so, all we do is edit the scripts to use the build path, not /usr/src/...
cp $RPM_HELPERS_DIR/{symset-table,find-requires{,.ksyms}} .
FIND_REQUIRES="$(pwd)/find-requires"
chmod 755 {symset-table,find-requires{,.ksyms}}
local tmp="$(pwd)"
tmp="${tmp
ed find-requires <<EOF
1a
set -x
.
/|.*find-requires.ksyms/s/|/| bash -x/
g/ [^ ]*\/\(find-requires\.ksyms\)/swq
EOF
ed find-requires.ksyms <<EOF
1a
set -x
.
g/\/.*\/\(symset-table\)/sg/\(\/usr\/src\/kernels\/\)/swq
EOF
ed symset-table <<EOF
1a
set -x
.
g/\(\/boot\/\)/sg/\(\/usr\/src\/kernels\/\)/swq
EOF
|
|
I saw a refresh of this patch. Does this mean the issues of the Intel build farm have been resolved?
|
|
There were just a large number of conflicts piling up so I did a refresh. No other changes, so any outstanding problems are probably still there.
|
|
somehow build 38351 did not produce any kmod-lustre rpm
|
|
Could you be more specific? It looks to like the the rhel7 builder for build 38351 produced kmode-lustre rpms:
|
|
Oh, they were produced, but it looks like lbuild failed to copy them? Hmm, did I drop some change from lbuild in the rebase? I'll look into that.
|
|
Oh, whew, I didn't change anything by accident. But the "LU-7887 lbuild: use "make rpms" for building Lustre RPM" made an unfortunate lbuild change that we now have to deal with. I'll refresh the patch some time today to address that.
|
|
It looks like patch set 30 dealt with the LU-7887 change just fine.
But testing is all failing.
|
|
Minh, it looks like there are problems at package installation time. In one of the logs named "node-provisioning-1.node-provisioning_1.autotest.trevis-38vm1.log" I am seeing the following failure:
First of all, the scripts are failing to detect the failure of the installation process. The provisioning step is declared a success even though it it failing.
Next, the scripts appear to be trying to install lustre-osd-ldiskfs which is not a valid package name under this patch. We probably need to update the build farm scripts to understand the new package names.
|
|
got it. I'll check it out
|
|
actually the error was
--> Finished Dependency Resolution
Error: Package: kmod-lustre-osd-ldiskfs-2.8.52_37_g9d5323c-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)
Requires: ksym(kill_block_super) = 0x0f72d0ce
Error: Package: kmod-lustre-osd-ldiskfs-2.8.52_37_g9d5323c-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)
Requires: ksym(jbd2_journal_destroy) = 0xb5132dfa
|
|
we need to rebase due to a bug in LU-7887
|
|
with the latest build yum install kmod-lustre-osd-zfs still failed dependencies. I have install zfs, kmod-zfs, spl, ...
---> Package lustre-osd-zfs-mount.x86_64 0:2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64 will be installed
--> Finished Dependency Resolution
Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)
Requires: ksym(dmu_objset_disown) = 0xde76f7a7
Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)
Requires: ksym(sa_lookup) = 0x9fec99c4
Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)
Requires: ksym(spa_writeable) = 0xd3eb11e1
Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)
Requires: ksym(zap_add_uint64) = 0xfe7bbafd
Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)
here is what I have on the node
[root@trevis-28vm2 yum.repos.d]# rpm -qa | grep lustre
lustre-osd-ldiskfs-mount-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64
lustre-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64
kernel-headers-2.6.32-573.22.1.el6_lustre.x86_64
kmod-lustre-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64
kmod-lustre-osd-ldiskfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64
lustre-tests-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64
kmod-spl-2.6.32-573.22.1.el6_lustre.x86_64-0.6.4.2-1.el6.x86_64
kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64-0.6.4.2-1.el6.x86_64
kernel-2.6.32-573.22.1.el6_lustre.x86_64
kernel-devel-2.6.32-573.22.1.el6_lustre.x86_64
lustre-iokit-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64
kernel-firmware-2.6.32-573.22.1.el6_lustre.x86_64
[root@trevis-28vm2 yum.repos.d]#
|
|
OK, so find the rpm for the kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64-0.6.4.2-1.el6.x86_64 packge. Run this command on it:
What symbol version does it say that dmu_objest_disown has in that package?
If it is anything other than "0xde76f7a7", that will explain why you are getting this error. If the line matches, then we'll move on to other problems.
|
|
right, that's the issue. kmod-zfs doesn't provide any symbol.
- rpm -qp --provides ./kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64-0.6.4.2-1.el6.x86_64.rpm
kernel-modules-for-kernel = 2.6.32-573.22.1.el6_lustre.x86_64
kmod-zfs-uname-r = 2.6.32-573.22.1.el6_lustre.x86_64
zfs-kmod = 0.6.4.2-1.el6
kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64 = 0.6.4.2-1.el6
kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64(x86-64) = 0.6.4.2-1.el6
|
|
Great! That seems like the next Intel buildfarm problem that you need to tackle.
Let me know if I can help in any way.
|
|
I look at http://zfsonlinux.org/generic-rpm.html and notice the steps to build zfs mod as below
$ cd spl-x.y.z
$ ./configure
$ make rpm-utils rpm-kmod
- Install the spl packages, they are required to build zfs.
$ sudo yum localinstall \
kmod-spl-devel-x.y.z-r.dist.arch.rpm \
kmod-spl-devel-kernel-x.y.z-r.dist.arch.rpm \
kmod-spl-kernel-x.y.z-r.dist.arch.rpm \
spl-x.y.z-r.dist.arch.rpm
$ cd ../zfs-x.y.z
$ ./configure
$ make rpm-utils rpm-mod
and we are using rpm build. I'll start there
|
|
Chris, I have repeated the build manually, (ie not using lbuild) and it still failed the dependency. I don't think this is lbuild issue but building kmod-lustre-osd-zfs with kmod-zfs issue
|
|
You are saying that the kmod-zfs package has no ksym symbols under the Provides even when you build it yourself?
Alright, can you step me through the details of how you built zfs manually?
|
|
here are the steps
wget http://archive.zfsonlinux.org/downloads/zfsonlinux/spl/spl-0.6.4.2.tar.gz
wget http://archive.zfsonlinux.org/downloads/zfsonlinux/zfs/zfs-0.6.4.2.tar.gz
tar zxvf spl-0.6.4.2.tar.gz
cd spl-0.6.4.2
./autogen.sh
./configure
make rpm-utils rpm-kmod
install all spl rpm
tar zxvf zfs-0.6.4.2.tar.gz
cd zfs-0.6.4.2
./autogen.sh
./configure
make rpm-utils rpm-kmod
then clone and checkout the patch
sh autogen.sh
./configure --with-zfs=/usr/src/zfs-0.6.4.2/ --with-spl=/usr/src/spl-0.6.4.2/ --without-ldiskfs
make rpm
|
|
What distro is that on? What kernel version?
What is the output of "rpm -qp --provides" for the kmod-zfs package that results?
|
|
[root@onyx-21vm4 lustre-release]# uname -a
Linux onyx-21vm4.onyx.hpdd.intel.com 2.6.32-573.22.1.el6.x86_64 #1 SMP Wed Mar 23 03:35:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@onyx-21vm4 lustre-release]#
[root@onyx-21vm4 zfs-0.6.4.2]# rpm -qp --provides kmod-zfs-2.6.32-573.22.1.el6.x86_64-0.6.4.2-1.el6.x86_64.rpm
kernel-modules-for-kernel = 2.6.32-573.22.1.el6.x86_64
kmod-zfs-uname-r = 2.6.32-573.22.1.el6.x86_64
zfs-kmod = 0.6.4.2-1.el6
kmod-zfs-2.6.32-573.22.1.el6.x86_64 = 0.6.4.2-1.el6
kmod-zfs-2.6.32-573.22.1.el6.x86_64(x86-64) = 0.6.4.2-1.el6
[root@onyx-21vm4 zfs-0.6.4.2]#
|
|
Minh, you are going to want to add the configure option "--with-spec=redhat" when building both spl and zfs for the centos/rhel systems. That will get you the packages with the needed symbol dependencies.
|
|
During my investigation of the previous package installation problem, I found another problem. Take a look at the postinstall script from the kmod-lustre-osd-zfs package generated under RHEL6.7:
Note the path for the find command. It is looking in /lib/modules/2.6.32-573.26.1.el6.x86_64/extra/lustre-osd-zfs for the module, but that directory doesn't exist.
I think that the simplest solution would probably just to make those subdirectories and move the modules in the %install section of the spec file. I'll give that a try.
|
|
FYI, I am about to push another update to the patch adding a couple of more Obsoletes entries.
|
|
Chris, have you (or anyone) tried to install kmod-lustre-osd-zfs from your own build?
|
|
OK, patch set 38 addresses every issue that I know about in the patch. I tested it through package installation on a RHEL6.7 VM, and all went well. The dependencies on the zfs package all matched up, and the symlinks from all of the modules looked like they were created correctly, even for the osd modules (that wasn't working on RHEL until one of my updates last week).
This LU-5614 patch together with the patch from LU-7643 are already allowing Lustre to build cleanly under LLNL's local koji build farm. That is a really nice improvement.
The only problem that I know of now is that we need to get a version of ZFS built in Intel's build far that lists all of the provided symbols in the package Provides: section. We know one way to fix that (add --with-spec=redhat to the configure line on RHEL-like systems).
Barring undiscovered issues, I think the patch is pretty darn close to complete.
|
|
that's great news!
I am curious as to why zfs configure script does not detect RHEL-like systems but have to manually specified by users
|
|
Chris, I noticed when we use 'generic' spec, we produced two kmod-spl-devel
kmod-spl-devel-0.6.4.2-1.el6.x86_64.rpm
kmod-spl-devel-2.6.32-573.22.1.el6.x86_64-0.6.4.2-1.el6.x86_64.rpm
but on 'redhat' spec, we only have one
kmod-spl-devel-0.6.4.2-1.el6.x86_64.rpm
|
|
I don't think there is anything to be concerned about there.
|
|
is kmod-spl-devel-0.6.4.2-1.el6.x86_64.rpm the same between the 'generic' and 'redhat' spec?
|
|
You can always rpm -qpl the package to find out. My guess is that there are differences in the contents. Why is this an issue?
|
|
I am still looking into this. the 'redhat' spec is building against the builder's kernel, not the one we want for lustre.
|
|
Chris, it seems to me that redhat/*-kmod.spec does not allow to build any kernel other than the one in /usr/src
%define ksrc %{_usrsrc}/kernels/%
{kverrel}
in order to use it for lustre, we need to install lustre kernel onto the builder after we patched the kernel.
any advice?
|
%define ksrc %{_usrsrc}/kernels/%
You should be able to override the %{_usrsrc}} variable (either at the build invocation time, or in ~/.rpmmacros) value to point at where you unpacked your kernel no?
|
|
Yeah, getting the Intel buildfarm to do things normally (actually install the packages that are BuildRequires of lustre) is probably not going to happen any time soon. So in the mean time we are probably stuck with various variable overrides.
I don't think that overriding _usrsrc will work on its own because the zfs "redhat" zfs-kmod.spec file uses %kernel_module_package, and that will always look instead for a properly installed kernel. That can be overriden, but that will take modifications to the zfs-kmod.spec script. I am none too certain that the zfs community is going to accept patches that dirty up their spec file for a completely non standard build environment.
So maybe we should go back to the "generic" zfs-kmod.spec file. The "generic" spec file is kmods 2 based and designed to work well in build environments like rpmfusion. It sounds like that spec file was easier to build in the Intel buildfarm. The problem with those packages is that they were failing to list Provides for the provided kernel symbols. I verified that issue on a clean RHEL6.7 image. The zfs community is going to be alot more receptive to allowing a change to add those Provides.
So I think we should put some effort into figuring out how to add the kernel symbol Provides to the generic zfs spec files.
|
|
yes, I am able to build spl with spl-kmod.spec modified; only a few lines like below
31,34d30
< %if !%{defined kernels}
< %define kernels %{kverrel}
< %endif
<
37d32
< %if !%{defined ksrc}
39d33
< %endif
51d44
<
104c97
< %{__rm} -f %{buildroot}/lib/modules/%{kernels}/modules.*
---
> %{__rm} -f %{buildroot}/lib/modules/%{kverrel}/modules.*
106c99
< %{__chmod} u+x %{buildroot}/lib/modules/%{kernels}/extra*
---
> %{__chmod} u+x %{buildroot}/lib/modules/%{kverrel}/extra*
112d104
< /lib/modules/%{kernels}/extra*
If you think zfs community won't allow such change for flexibility, then I will pursue the 'generic' spec
|
|
Just a quick tip: always use unified diff format: "diff -u". A decent set of standard options would be "diff -uNp". Better yet, use git. The "git diff" command does the right thing by default.
|
|
update: latest version with --define "_use_internal_dependency_generator 0" added worked!!
I have verified osd-zfs loaded with kmod-zfs and lustre mounted on the server. client can move to later kernel without problem.
I am proceeding to investigate why maloo failed
|
|
Chris, I want to review the packages and provides; just to make sure
[root@onyx-21vm7 build_type-server_distro-el6.7_arch-x86_64_ib_stack-inkernel]# rpm -qp --provides lustre-dkms-2.8.52_70_g3660338-1.el6.noarch.rpm
lustre-kmod = 2.8.52_70_g3660338
lustre-modules = 2.8.52_70_g3660338
lustre-osd
lustre-osd-zfs = 2.8.52_70_g3660338
lustre-dkms = 2.8.52_70_g3660338-1.el6
should lustre-dkms also provides lustre-osd-ldiskfs?
|
|
I have never reviewed what they did in the dkms packaging.
|
|
I am investigating an sles12sp1 build failure that could be from this patch
https://build.hpdd.intel.com/job/lustre-reviews/39104/arch=x86_64,build_type=server,distro=sles12,ib_stack=inkernel/consoleText
d4d54bfba3e5f20ab369aa69b0f9b6dcfab1cdce debuginfo(build-id) = d8c98b864c6e0ac2360a32da503bd15d7cbbc263 debuginfo(build-id) = da62063a63187395fb3c9b3aae6e3541ed005684 debuginfo(build-id) = da697fd1e5187bf2f6ddc0dc1528c9050b51f08d debuginfo(build-id) = e46c8a226902868165db3894ee0450d05be00325 debuginfo(build-id) = f6cfd424f753677f8f61b05c6326af88cdafc5da debuginfo(build-id) = f8134d77ab4d46f6c4cd994efab8518e9e8b8372 lustre-tests-debuginfo = 2.8.53_27_ga8b7e8c-3.12.57_60.35_lustre_default lustre-tests-debuginfo(x86-64) = 2.8.53_27_ga8b7e8c-3.12.57_60.35_lustre_default
Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Checking for unpackaged file(s): /usr/lib/rpm/check-files /tmp/rpmbuild-lustre--lud8gVUp/BUILDROOT/lustre-2.8.53_27_ga8b7e8c-1.x86_64
error: Installed (but unpackaged) file(s) found:
/lib/modules/3.12.57-60.35_lustre-default/extra/lustre-osd-ldiskfs/fs/ldiskfs.ko
/lib/modules/3.12.57-60.35_lustre-default/extra/lustre-osd-ldiskfs/fs/osd_ldiskfs.ko
/lib/modules/3.12.57-60.35_lustre-default/extra/lustre-tests/fs/llog_test.ko
/lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/fid.ko
/lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/fld.ko
/lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/lfsck.ko
/lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/llite_lloop.ko
/lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/lmv.ko
|
|
It looks like there were problems with node provisioning step in the most recent revision of the patch. Are you working on a solution for that too?
|
|
Yes, I am. I will make sure this patch get tested
|
|
the latest issue is on zfs el7, can't load osd-zfs
May 29 08:04:34 onyx-42vm3 mrshd[7705]: root@onyx-42vm5.onyx.hpdd.intel.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/mds1; mount -t lustre #011#011 lustre-mdt1/mdt1 /mnt/mds1");echo XXRETCODE:$?'
May 29 08:04:37 onyx-42vm3 kernel: LNet: HW CPU cores: 2, npartitions: 1
May 29 08:04:37 onyx-42vm3 kernel: alg: No test for adler32 (adler32-zlib)
May 29 08:04:37 onyx-42vm3 kernel: alg: No test for crc32 (crc32-table)
May 29 08:04:46 onyx-42vm3 kernel: Lustre: Lustre: Build Version: 2.8.53_28_g7808b96
May 29 08:04:46 onyx-42vm3 kernel: LustreError: 158-c: Can't load module 'osd-zfs'
May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(genops.c:318:class_newdev()) OBD: unknown type: osd-zfs
May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(obd_config.c:370:class_attach()) Cannot create device lustre-MDT0000-osd of type osd-zfs : -19
May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(obd_mount.c:198:lustre_start_simple()) lustre-MDT0000-osd attach error -19
May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(obd_mount_server.c:1809:server_fill_super()) Unable to start osd on lustre-mdt1/mdt1: -19
May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(obd_mount.c:1450:lustre_fill_super()) Unable to mount (-19)
May 29 08:04:46 onyx-42vm3 xinetd[915]: EXIT: mshell status=0 pid=7704 duration=12(sec)
May 29 08:04:46 onyx-42vm3 systemd-logind: Removed session c35.
I have checked manually with el6.7 but not el7. I'll check on el7
|
|
the reason is on el7, somehow another 'x86_64' string was added in the path
D: %post(kmod-lustre-osd-zfs-2.8.53_28_g7808b96-3.10.0_327.13.1.el7_lustre.x86_64.x86_64): scriptlet start
D: %post(kmod-lustre-osd-zfs-2.8.53_28_g7808b96-3.10.0_327.13.1.el7_lustre.x86_64.x86_64): execv(/bin/sh) pid 25714
+ '[' -e /boot/System.map-3.10.0-327.13.1.el7_lustre.x86_64.x86_64.x86_64 ']'
+ modules=($(find /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64.x86_64.x86_64/extra/lustre-osd-zfs | grep '\.ko$'))
++ grep '\.ko$'
++ find /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64.x86_64.x86_64/extra/lustre-osd-zfs
find: ‘/lib/modules/3.10.0-327.13.1.el7_lustre.x86_64.x86_64.x86_64/extra/lustre-osd-zfs’: No such file or directory
|
|
Can you please provide a pointer to the failed test in maloo?
|
|
actually I had to run rpm manually to get that error message. I thought your patch from LU-7643 would solve it but it still failed
|
|
I wonder if this is related to kmodtool. I think zfs/spl is using a later version than the one provided with the distro. should lustre do the same?
|
|
It does not surprise me that LU-7643 does not help.
With a stock rhel7.2 kernel with zfs, and I cannot reproduce this problem. So I am guessing that this is another Intel build system problem.
I think the first thing I would try is adding a %{warn} statement to the line after "%{!?kernel_version: %global kernel_version %kversion}" in the lustre spec file to print out the value of %kernel_version. That warn statement might show the weird value right there, and then you can work backward from that to find where it originates.
|
|
yeah, so the kernel_version print out is "3.10.0-327.18.2.el7.x86_64"
This sounds like we need to build lustre kmod using kmods 2 (similar to spl/zfs using generic spec)
|
|
I think it is too early to give up on this approach. Lets keep digging.
|
|
On RHEL7, /usr/lib/rpm/redhat/kmodtool has the following function:
get_kernel_release ()
{
if [[ -z $1 ]]; then
uname -r
return
fi
local arch=$(arch)
local verrel=${1%.$arch}
local verprefix=${verrel%.*}
local versuffix=${verrel#$verprefix}
verrel=$(ls -Ud /usr/src/kernels/$verprefix*$versuffix.$arch | sort -V | tail -n 1)
verrel=${verrel##*/}
[[ -z $verrel ]] && verrel=$1.$arch
echo "$verrel"
}
This is no doubt where the problem arises for lbuild on rhel7. lbuild, as we all know, employees a very non-standard build environment. Since the kernel is not in the correct location during packaging, this function is falling back to setting verrel to "$1.$arch". That is almost certainly where that extra ".x86_64" is coming from, but only under lbuild on rhel7.
The trick here will be to come up with more hackery for lbuild that allows the lustre.spec file to remain relatively clean for proper build environments.
|
|
Meanwhile, Minh, do you have a plan to fix the Intel buildfarm to install the correct packages when this patch is in use? The node provisioning step still seems to always install the old packages, which winds up pulling in part kmod and part dkms packages.
|
|
Chris, actually I have fixed the one under Onyx cluster. We are rolling out to the rest of the clusters. I checked and the installation has worked on Onyx. Unfortunately, most of the testing went to Trevis cluster in the last few days (week) due to Onyx being loaded with master testing.
|
|
OK, great. If you have that figured out, then we could be in good shape.
I believe that I have hacked around lbuild's latest problem on RHEL7. I tested my hack in http://review.whamcloud.com/20572, and it seemed to work. No triple ".x86_64" in the scripts, and it passed testing...but that was on trevis. I don't know why it passed there despite failing ldiskfs package installation. Very strange.
I squashed my work-around into Patch Set 44 of this ticket's patch, http://review.whamcloud.com/12063.
Are there any other known problems?
|
|
actually, I also have a version and I am testing right now. http://review.whamcloud.com/#/c/18426/
Please take a look while I continue to test
|
|
The patch is apparently hung up on unrelated test failures now.
It would still be good to get reviews in now, so we can get this landed as soon as reasonable.
|
|
I found in build log:
+ echo 'The kernel ABI reference files (provided by kabi-whitelists) were not found.'
The kernel ABI reference files (provided by kabi-whitelists) were not found.
+ echo 'No compatibility check was performed. Please install the kABI reference files'
No compatibility check was performed. Please install the kABI reference files
+ echo 'and rebuild if you would like to verify compatibility with kernel ABI.'
and rebuild if you would like to verify compatibility with kernel ABI.
+ echo ''
we need to install kabi-whitelists rpm don't we?
|
|
It helps, but Lustre uses symbols not the in the whitelist, so you will still get a warning.
|
|
Does it still?  I know this was the case when I tried to make all of this work with RHEL many years ago.
|
|
I don't this we'll get the warning after install the kabi-whitelist rpm. I did it locally and didn't see that. we'll update the builders and see
|
|
I just tried the latest patch and I don't see lustre-module-* anymore. Is this expected?
I also tried installing them RPMs and got:
rpm Uvh --force tmp/RPMS/lustre/lustre*
error: Failed dependencies:
kmod-lustre = 2.8.54_60_g2a55f34_dirty is needed by lustre-2.8.54_60_g2a55f34_dirty-2.6.32_573.12.1.el6.head.x86_64.x86_64
kmod-lustre = 2.8.54_60_g2a55f34_dirty is needed by lustre-tests-2.8.54_60_g2a55f34_dirty-2.6.32_573.12.1.el6.head.x86_64.x86_64
|
|
James,
If you carefully read the commit header it says how the packages have been renamed.
|
|
Thank you for picking this up and following through with it.
My complaint is that I'm still the owner and can't give you a +1!
Edit: I won't bother nitpicking changes to lbuild.
|
|
Okay, for some reason my initial build didn't produce the kmod-* rpms. I had to delete the source tree and do it over again. Now I see
ksym(snprintf) = 0x9edbecae is needed by kmod-lustre-2.8.54_60_g2a55f34_dirty-2.6.32_573.12.1.el6.head.x86_64.x86_64
ksym(sscanf) = 0x42224298 is needed by kmod-lustre-2.8.54_60_g2a55f34_dirty-2.6.32_573.12.1.el6.head.x86_64.x86_64
....
When I go to install kmod-lustre-*.x86_64.rpm
|
|
James, are you able to build and install it. Please let me know if you have any problem. if not could you re-review that patch?
|
|
Nope, but looking at how I build our kernel I might know why. When building the kernel rpm I used --without kabichk. Would that be the reason it doesn't work? This is all done using RHEL6.7.
|
|
Remove --without kabichk and still gives me the ksym issues.
|
|
Is the target kernel installed on the build system (or environment)?
Can you verify that dependency generation is done with the right kernel?
|
|
No the target kernel rpm is not installed on the build system. That would be a big no no since the machine is used for other purposes. The target kernel source tree is installed in a special directory which is not /usr/src to build against. I will be looking into why the dependency generation is done against the wrong kernel. I wonder if we have to do a OFED style build process in which we tell the location of the symver.
|
|
So I created the most basic test to show the problem. This is with building just the patchless client. Currently our build machine is at RHEL6.7 and my test node is running RHEL6.8. So we have very different kernels running on both. On the build machine I as non-root installed the RHEL6.8 development tree in /tmp so I have /tmp//usr/src/kernels/2.6.32-642.1.1.el6.x86_64. Then I went into my lustre tree containing this patch and did:
cd lustre_release
./autogen.sh
./configure --disable-server --with-linux=/tmp/usr/src/kernels/2.6.32-642.1.1.el6.x86_64
make rpm
Then I went to install the rpms into the image and got the ksyms errors still. This points to the current patch for LU-5614 breaks --with-linux for configure. Note this is very common process our admins use to prepare our client rpms. You can say well just build on the local node. In that case the patch for LU-5614 need to be updated to remove --with-linux support. Also if you recommend using mock to work around this problem then please update this patch to ensure configure failed unless ./configure runs in a mock environment and the source rpm has a hard dependency on mock. In any case I have a reproducer of this problem.
|
|
it worked on el7. I'll verify on el6 as you did next
[root@onyx-21vm5 lustre-release]# uname -a
Linux onyx-21vm5.onyx.hpdd.intel.com 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
sh ./autogen.sh
./configure --disable-server --with-linux=/usr/src/kernels/3.10.0-327.18.2.el7.x86_64/
make rpms
scp kmod-lustre-client-2.8.54_61_gcc7a8c9-3.10.0_327.18.2.el7.x86_64.x86_64.rpm root@onyx-24:/root
scp lustre-client-2.8.54_61_gcc7a8c9-3.10.0_327.18.2.el7.x86_64.x86_64.rpm root@onyx-24:/root
[root@onyx-24 ~]# uname -a
Linux onyx-24.onyx.hpdd.intel.com 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@onyx-24 ~]# rpm -hiv ./kmod-lustre-client-2.8.54_61_gcc7a8c9-3.10.0_327.18.2.el7.x86_64.x86_64.rpm
Preparing... ################################# [100%]
Updating / installing...
1:kmod-lustre-client-2.8.54_61_gcc7################################# [100%]
[root@onyx-24 ~]# modprobe lustre
[root@onyx-24 ~]# modinfo lustre
filename: /lib/modules/3.10.0-327.18.2.el7.x86_64/extra/lustre-client/fs/lustre.ko
license: GPL
version: 2.8.54_61_gcc7a8c9
description: Lustre Client File System
author: OpenSFS, Inc. <http://www.lustre.org/>
rhelversion: 7.2
srcversion: F6CD34A134CF6E45A10B0DB
depends: obdclass,ptlrpc,libcfs,lnet,lmv,mdc,lov
vermagic: 3.10.0-327.18.2.el7.x86_64 SMP mod_unload modversions
[root@onyx-24 ~]#
|
|
Okay. Lets see if its a RHEL6.X issue. Especially since I don't have access to RHEL7 systems If you manage to build it then the build process must be dependent on something missing on our build machine. If I see this problem other people will to once its released into the wild.
|
|
I don't see any issue on el6. I built on el6.7 pointing to el6.8 kernel and install kmod lustre on el6.8 system. no issue
[root@onyx-21vm2 lustre-release]# uname -a
Linux onyx-21vm2.onyx.hpdd.intel.com 2.6.32-573.26.1.el6.x86_64 #1 SMP Wed May 4 00:57:44 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
sh autogen.sh
./configure --disable-server --with-linux=/usr/src/kernels/2.6.32-642.1.1.el6.x86_64/
make rpms
scp kmod-lustre-client-2.8.54_61_gcc7a8c9-2.6.32_642.1.1.el6.x86_64.x86_64.rpm onyx-21vm3:/root
scp lustre-client-2.8.54_61_gcc7a8c9-2.6.32_642.1.1.el6.x86_64.x86_64.rpm onyx-21vm3:/root
history
rpm -hiv ./kmod-lustre-client-2.8.54_61_gcc7a8c9-2.6.32_642.1.1.el6.x86_64.x86_64.rpm
[root@onyx-21vm3 ~]# modprobe lustre
LNet: HW CPU cores: 1, npartitions: 1
alg: No test for adler32 (adler32-zlib)
alg: No test for crc32 (crc32-table)
Lustre: Lustre: Build Version: 2.8.54_61_gcc7a8c9
LNet: Added LNI 10.2.4.14@tcp [8/256/0/180]
LNet: Accept secure, port 988
[root@onyx-21vm3 ~]# modinfo lustre
filename: /lib/modules/2.6.32-642.1.1.el6.x86_64/extra/lustre-client/fs/lustre.ko
license: GPL
version: 2.8.54_61_gcc7a8c9
description: Lustre Client File System
author: OpenSFS, Inc. <http://www.lustre.org/>
srcversion: F6CD34A134CF6E45A10B0DB
depends: obdclass,ptlrpc,libcfs,lnet,lmv,mdc,lov
vermagic: 2.6.32-642.1.1.el6.x86_64 SMP mod_unload modversions
[root@onyx-21vm3 ~]# uname -a
Linux onyx-21vm3.onyx.hpdd.intel.com 2.6.32-642.1.1.el6.x86_64 #1 SMP Tue May 31 21:57:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@onyx-21vm3 ~]#
I am not sure what's missing. I setup the builder without anything special. installed git, libtool, rpm-build
How did you put the kernel-devel on the builder? cpio or rpm install?
|
|
Are you ssh into the destination node and then installing the rpm on that node? For our systems we create the rpms and chroot into a directory that serves as the root of my image for the diskless test nodes.
[ management server ] -> /export-image/root -> diskless-node:/root
/usr /usr
chroot /export-image
rpm -ivh /tmp/lustre.rpm
Could the chroot environment be causing the ksym issues? In our diskless setup its not really possible to install rpms directly on the test nodes since they are essentially read only. Can you try that setup Minh please.
|
|
"Are you ssh into the destination node and then installing the rpm on that node?"
yes
I am not familiar with chroot. For diskless node, I know that LLNL Chaos built the image with the set of rpms, then refresh/boot the node. I'll have to look into chroot more; but I am pretty sure that's the difference and causing the issue.
Can you attach or upload your kmod-lustre-client rpm so I can check its content?
|
|
I found the source of the problem. This change now requires the package kabi-whitelist at least for RHEL6. Do you have this package for RHEL7 and SLES as well? Once I installed kaki-whitelist it appears to work. Well ZFS still gives trouble but it might be a simple case of rebuilding it.
|
|
I'm back from work travel, so I can jump back into the conversation now.
James, I don't believe that kernel-abi-whitelists is required on either RHEL6 or RHEL7. Yes, there will be a warning about it being missing at build time, but that doesn't hurt the packaging process. Clean RHEL6.7 and RHEL7.2 images have been my main testing platforms (with some SLES 12.X on occasion). I checked both of my RHEL images just now, and neither have kernel-abi-whitelists installed. The rpm packages generated under this patch install just fine, and appear to have all required kernel symbols expressed in the rpm --requires section.
By the way, if your packages really lists this:
then there is an additional problem that I didn't remember off the top of my head on the road. Those should not be "ksym" requirements; they should be "kernel" requirements. I would check the package with "rpm -qp --requires" and see how they look. Here are the requirements from the kmod-lustre package on my RHEL6.7
system:
A "kernel" requirement on RHEL means that the RH build system knows that the symbol is provided by the kernel package itself. Any other kernel symbol provided by some external package (e.g. zfs) will be a "ksym" requirement instead of "kernel". So this still implies that Lustre was not built on a system with the prerequisite kernel rpms properly installed.
FYI, SUSE does not have a kernel-abi-whitelists package. They do not even employ the approach of having a symbol whitelist.
https://drivers.suse.com/doc/SolidDriver/SUSE_Kernel_ABI_Stability.html#does-suse-provide-a-kernel-abi-symbol-whitelist
|
|
Minh said:
we need to install kabi-whitelists rpm don't we?
That is not a requirement for this ticket. It probably won't hurt, but you can track that activity in a separate ticket if you like. It does not need to be linked to this issue.
|
|
In reply to James from gerrit:
I don't modify any RHEL build scripts. Its a simple rpm extract source and build. Nothing fancy. Forcing people to require things like mock to just build client rpms is not going to go over well with sites.
No one is requiring special things like mock. We just require that, when building rpms, the prerequisite BuildRequires must actually be installed. This is a completely normal thing to expect in the rpm packaging world, and it is what all of the behind-the-scenes scripts from the distro assume will be the case. Most people will be able to meet that requirement, I think. To avoid installing the prerequisites, you have to replace system-provided behind-the-scenes scripts with your own. We did that for lbuild, despite how icky I felt doing it. . If you refuse to meet that simple, accepted requirement, you are free to hack up your own solution as well.
I just recommend mock for the places where you don't have access to a system where the BuildRequires are installed, or you don't have the permissions to install those packages. mock is designed exactly for those situations: it sets up a clean base chroot environment, and then installs only the packages required by the source rpm's BuildRequires.
If none of those things will work for people, they can always just download prebuilt binaries. The prebuilt binaries are going to work on a much wider set of kernels once they are no longer tied to one specific kernel version.
Yes, some adaptation of process will be necessary for some people. I know, change is hard. I totally feel your pain. The Lustre community has gotten very used to its oddball way of building and packaging. But that means that Lustre just flat out doesn't build through the standard tools of major distros like Fedora and RHEL. I think the advantage of supporting those standard methods vastly outweighs the short term pain we will feel as we adapt our personal processes to the new packaging methods.
|
|
I discovered the kabi-whitelist issues with the following error during the build process:
Finding Provides: /usr/lib/rpm/redhat/find-provides
KERNEL ABI COMPATIBILITY WARNING
The kernel ABI reference files (provided by kabi-whitelists) were not found.
No compatibility check was performed. Please install the kABI reference files
and rebuild if you would like to verify compatibility with kernel ABI.
Finding Requires: /usr/lib/rpm/redhat/find-requires
Once I installed kabi-whitelish and the rpms started to work. Also 'rpm qp --release kmod-lustre*' prints out
kernel(__wake_up) = 0x642e54ac
kernel(add_wait_queue) = 0x650fb346
kernel(copy_from_user) = 0x3302b500
kernel(default_wake_function) = 0xffd5a395
kernel(kfree) = 0x037a0cba
So its working correct now.
|
|
Oleg Drokin (oleg.drokin@intel.com) merged in patch http://review.whamcloud.com/12063/
Subject: LU-5614 build: use %kernel_module_package in rpm spec
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 3b7d27ea22faf1c6d0a37afa724fd9b5c3240322
|
|
The patch landed to master, so I closing this ticket. Good work everyone!
If there is any fallout that we need to work through, we can open new tickets for that.
|
|
yes, a small fix for suse12 is in LU-8343
|
|
FYI:
http://cdn.opensfs.org/wp-content/uploads/2016/04/LUG2016D1_Improved-Versioning_Morrone_DiNatale.pdf
https://www.youtube.com/watch?v=50VN9Q7BHYA&index=5&list=PLA5dHg1_l3V8r6eqrUnl8DiB-KH1K5yLH
|
Generated at Sat Feb 10 01:53:01 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
lustre-ldiskfs.files