https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5614 Lustre <p>The correct way to support weak-updates in rpm packages is the vendor defined %kernel_Module_package. This does the right thing on all distributions.</p> <p>We have used this feature in SGI Lustre for several years, and I plan to work this feature back into the master branch.</p> LU-5614

use %kernel_module_package for weak-updates

Bug Critical Closed Fixed Minh Diep Stephen Champion llnl patch Fri, 12 Sep 2014 04:15:18 +0000 Tue, 6 Jun 2017 15:37:53 +0000 Mon, 18 Jul 2016 21:54:05 +0000 Lustre 2.9.0 1 23 <p>Minh</p> <p>Could you please review these patches when Steve supplies them?</p> <p>Thanks</p> <p>Peter</p> <p>Don't hold your breathe waiting for them ;^)</p> <p>I started to take a look, and realized I need to have a handle on what lbuild does before I work this out.</p> <p>Can you just paste your complete <tt>lustre.spec</tt> here and we can see which side gets to it first? <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Simplified version of the SGI spec</p> <p>Anyone is very welcome to jump ahead of me.</p> <p>I just attached a stripped down version of the spec used for our 2.4.2 release. Essentially, the %kernel_module_package macro replaces the module package and all of the associated pre/post scripts. The tricky bit is that it expects to build for every installed kernel. You either have to override this or deal with it.</p> <p>There are also some hazards in the macro itself : ie, it expects that all installed kernels have the same version.</p> <p>All of this gets especially nasty dealing with the server packages, which is the default flavor with the version string twiddled (I dealt with this in SGI releases by building the server kernel as a distinct flavor). I suspect the answer is simple to not do weak-modules for servers, as there is limited benefit to it on servers.</p> <p>I was considering having a separate spec: either for clients vs servers, or for --with-weak-updates vs --without-weak-updates.</p> <p><a href="http://review.whamcloud.com/12063" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12063</a></p> <p>Intended only as a conversation piece at this point.</p> <p>^Build failed because there are multiple versions of the same kernel installed.</p> <p>%kernel_module_package normally uses rpm to query the installed kernel version, and packages modules for the first listed kernel version. That resulted in a mismatch between the kernel the modules were built for and the kernel the modules were (failed to be) packaged for.</p> <p>I'm exploring some ways around this.</p> <p>With RedHat, we can define kernel_version to override the default selection of which kernel version to build for. </p> <p>SuSE does not have anything like that, but it looks there is a mechanism with %suse_kernel_module_package (which is the underlying implementaion of %kernel_module_package on SLES). I will investigate that, and perhaps engage SuSE to get the SLES %kernel_module_package to use that facility in a way which is compatible with RH's.</p> <p>In the meantime, I'm looking at the build failure from <a href="http://review.whamcloud.com/#/c/12063/2" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12063/2</a>, which I could not reproduce. On my system, /usr/lib64/lustre/mount_osd_xyz.so is packaged with the osd xyz kmp. I suspect something odd about %</p> {with lustre_utils} <p>.</p> <p>Any further news on this one Stephen?</p> <p>Thanks,<br/> ~ jfc.</p> <p>Building on local systems with './configure ; make rpms', the patch I have is able to build for redhat when multiple kernel package versions are installed. It works on SLES with a single version installed.</p> <p>Building with Intel's jenkin's, these files:<br/> %{_libdir}/lustre/mount_osd_ldiskfs.so<br/> %{_libdir}/lustre/mount_osd_zfs.so</p> <p>which should be part of the their respective osd module packages, are not picked up, and I was unable to identify why this difference arises or how to address in it the time I had available.</p> <p>Alexander Boyko (alexander.boyko@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/14191" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14191</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> build: add kmod for client<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6748dbb146de7d8a8df72765996180a1a9b2837b</p> <p>I found that it needs to have kernel-devel installed on the builder. As we use lbuild to extract the kernel-devel on to the kdir, it doesn't seem to work</p> <p>++ /usr/bin/rpmbuild --target x86_64 -tb /mnt/build/lustre-release/lustre-2.7.55.tar.gz --without servers --define '__find_requires /mnt/build/lustre-release/BUILD/find-requires' --define 'configure_args --with-o2ib=no' --define 'kdir /mnt/build/lustre-release/BUILD/reused/usr/src/kernels/2.6.32-504.23.4.el6.x86_64' --define '_tmppath /var/tmp' --define '_topdir /mnt/build/lustre-release/BUILD'<br/> error: Failed build dependencies:<br/> kernel-devel is needed by lustre-client-2.7.55-2.6.32_504.23.4.el6.x86_64_ge550f31.x86_64</p> <p>lbuild and the environment it operates in is sufficiently nonstandard for host and rpm builds that we'll have difficulty getting it to use any of the standard kernel module package methods without addressing some of it's deficiencies.</p> <p>multiple kernel-devel packages can be installed concurrently, so there is no reason not to install the package if it is required by a build - and it is correctly required by any package building kernel modules.</p> <p>That said... The Buildreq is automatically added by %kernel_module_package. It looks like it can be overridden with nobuildreqs=yes.</p> <p>even with your latest patch, it won't install without building with kernel-devel</p> <p><span class="error">&#91;root@onyx-24 x86_64&#93;</span># rpm -hiv kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64.rpm<br/> error: Failed dependencies:<br/> ksym(__init_waitqueue_head) = 0xffc7c184 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(__mutex_init) = 0x4bf79039 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(__stack_chk_fail) = 0xf0fdf6cb is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(__wake_up) = 0x642e54ac is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(add_wait_queue) = 0x650fb346 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(copy_from_user) = 0x3302b500 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(default_wake_function) = 0xffd5a395 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(kfree) = 0x037a0cba is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64</p> <blockquote> <p>I found that it needs to have kernel-devel installed on the builder. As we use lbuild to extract the kernel-devel on to the kdir, it doesn't seem to work<br/> ++ /usr/bin/rpmbuild --target x86_64 -tb /mnt/build/lustre-release/lustre-2.7.55.tar.gz --without servers --define '__find_requires /mnt/build/lustre-release/BUILD/find-requires' --define 'configure_args --with-o2ib=no' --define 'kdir /mnt/build/lustre-release/BUILD/reused/usr/src/kernels/2.6.32-504.23.4.el6.x86_64' --define '_tmppath /var/tmp' --define '_topdir /mnt/build/lustre-release/BUILD'<br/> error: Failed build dependencies:<br/> kernel-devel is needed by lustre-client-2.7.55-2.6.32_504.23.4.el6.x86_64_ge550f31.x86_64</p></blockquote> <p>I think you can use <tt>--nodeps</tt> with <tt>rpmbuild</tt>. This is of course something that needs to be done with care and attention to not paper over other missing dependency requirements. But those will float up to the surface pretty quickly.</p> <blockquote> <p>even with your latest patch, it won't install without building with kernel-devel<br/> [root@onyx-24 x86_64]# rpm -hiv kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64.rpm<br/> error: Failed dependencies:<br/> ksym(__init_waitqueue_head) = 0xffc7c184 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(__mutex_init) = 0x4bf79039 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(__stack_chk_fail) = 0xf0fdf6cb is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(__wake_up) = 0x642e54ac is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(add_wait_queue) = 0x650fb346 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(copy_from_user) = 0x3302b500 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(default_wake_function) = 0xffd5a395 is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64<br/> ksym(kfree) = 0x037a0cba is needed by kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64</p></blockquote> <p>These are not missing <em>kernel-devel</em> dependencies. These are missing "weak-module" ABI dependencies.</p> <p>Which kernel(s) were on the machine you tried to install <em>kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64.rpm</em> on? The above errors are telling you that you don't have a kernel that is ABI (weak module) compatible with your <em>kmod-lustre-client-2.7.55-2.6.32_504.12.2.el6.x86_64_ge550f31.x86_64.rpm</em></p> <p>Minh or Steve,</p> <p>Just checking in to see where we are going with this ticket?</p> <p>Thanks,<br/> ~ jfc.</p> <p>Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/17489" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/17489</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> build: use %kernel_module_package in rpm spec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e36f89001ffc45710d8fd7dea44c944989018ef1</p> <p>I used a different Change-Id for my revision of Stephen's patch, and marked mine "fortestonly" to reduce confusion somewhat. Nothing too significantly different in mine yet, but I plan to continue work on mine next week. If folks like the changes and don't mind, I can update Stephen's original patch (if someone doesn't make the needed improvements before me).</p> <p>Stephen's work is an important step towards having Lustre sanely packed for RHEL. LLNL is moving to the koji build farm for our RHEL7-based TOSS3 distribution, and having this patch that gets weak-updates working is a high priority for us.</p> <p>Once we have this weak-modules support, we can finally drop the kernel version string from Lustre's release string. That will enable even more cleanup. Yay!</p> <p>I'm glad you are finding this work useful!</p> <p>You need to be careful about the build environment for the kmp packaging to work as expected. You can hack around limitations a bit, but really need to have only a single version-release installed. This was never an issue for my build system, but the Jenkins environment does not enforce it.</p> <p>To deal with some of the problems of existing installations, crossbuilds, etc, I was looking at having multiple rpm specs, selected by configure options.</p> <p>Christopher, I can't see my responses to your inline comments, so...</p> <p>You can do Obsoletes, Requires, etc with %kernel_module_package (%kmp) by providing a preamble. This a file which is expanded into the subpackage preamble. My use was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Source5: lustre-modules.files Source6: lustre-ldiskfs.files Source7: lustre-modules.preamble Source8: lustre-ldiskfs.preamble [ ... ] %kernel_module_package -p %SOURCE7 -n %{name} -f %SOURCE5 @TARGET_FLAVORS@ %kernel_module_package -p %SOURCE8 -n %{name}-ldiskfs -f %SOURCE6 @TARGET_FLAVORS@ lustre-modules.preamble: Provides: %{name}-modules %{name}-modules-%1 Obsoletes: %{old_kmod_obsoletes} %if %is_server Requires: lustre-backend-fs-%1 %endif lustre-ldiskfs.preamble: Requires: %{name}-modules-%1 Provides: lustre-backend-fs-%1 </pre> </div></div> <blockquote><p>%global kernel_version %kversion</p></blockquote> <blockquote><p>%kernel_module_package must want this variable set, but a comment to explain that would help avoid confusion</p></blockquote> <p>On RH based distros, it is used to override the default method of identifying which kernel to build for. It is not needed if the build environment is strictly controlled. See my Oct 1 comment.</p> <blockquote><p>Is there some reason not to simply use this line unconditionally?</p></blockquote> <p>You would need to move the without_servers condition into kmp-lustre.files<br/> That is probably a better way. As is, I think the client kmp is missing the %doc files</p> <blockquote> <p>Christopher, I can't see my responses to your inline comments, so...</p></blockquote> <p>You have to click the "review" button to post your inline comments.</p> <blockquote> <p>On RH based distros, it is used to override the default method of identifying which kernel to build for. It is not needed if the build environment is strictly controlled. See my Oct 1 comment.</p></blockquote> <p>Yes, I understand. I think you are right to include that for now, because changing the way Lustre selects which kernel to build against would only complicate the patch further. I am only asking that a comment be added to the spec file so that it is clearer to everyone in the future. You can see I added a comment in my version of the patch.</p> <blockquote> <p>To deal with some of the problems of existing installations, crossbuilds, etc, I was looking at having multiple rpm specs, selected by configure options.</p></blockquote> <p>I completely agree. Having one spec file seems like a nice idea at first, but once you get into all of the little details, it becomes clear that rpm just was not designed to support multiple distros (or really even multiple versions of the same distro) in one spec file. We really should switch to having many simpler spec files instead of one inscrutable and always broken spec file.</p> <p>I am not convinced yet whether configure should be involved. I would like to move towards having the build (compilation) system and the packaging system be more cleanly separated. But it is a possibility.</p> <p>Server packages build fine under CentOS7 with my version of the patch, but not in Intel's buildfarm. Sigh. Can we kill lbuild yet?</p> <p>I'd like to go back and answer John's question:</p> <blockquote><p>Just checking in to see where we are going with this ticket?</p></blockquote> <p>and I'll start by quoting Stephen Champion:</p> <blockquote><p>lbuild and the environment it operates in is sufficiently nonstandard for host and rpm builds that we'll have difficulty getting it to use any of the standard kernel module package methods without addressing some of it's deficiencies.</p></blockquote> <p>Stephen really hit the nail on the head. The nonstandard way that lbuild operates is the biggest obstacle to success in this ticket. It looks to me like Stephen's last revision of the patch was very close to complete. The remaining minor issues shouldn't take very long to work through, and I have already addressed some of them in my revision.</p> <p>Build systems are surprisingly complicated and difficult to get right for every revision of every distribution. Adding that inherent complication to the unnecessary complication of lbuild's non-standard approach leaves us with a problem that is practically insurmountable.</p> <p>I've been recommending that we ditch lbuild for something that uses more standard packaging approach for years now (<a href="https://jira.whamcloud.com/browse/LU-3956" title="Eliminate lbuild&#39;s nonstandard build process" class="issue-link" data-issue-key="LU-3956">LU-3956</a>, but I'd been advocating to change it long before I opened that ticket). I have not seen much effort in that area, unfortunately.</p> <p>So, Intel, how do we make progress here?</p> <p>I would really like to see this and other build and packaging improvements land early in the 2.9 landing window. Is there some way to address the problems in Intel's build farm in that time frame? I would assume that setting up a full lbuild replacement in just a month or two won't happen, but we'll need some attention to improve the current situation enough to let sane changes happen.</p> <p>So it would be good to lay out a plan for both short and longer term changes to Intel's buildfarm situation.</p> <p>Hi Chris</p> <blockquote><p>Server packages build fine under CentOS7 with my version of the patch, but not in Intel's buildfarm. Sigh. Can we kill lbuild yet?</p></blockquote> <p>I see that it passed EL6.7 but failed in EL7. Have you tried local build on EL7? If so please share the output log.</p> <p>I wonder what lbuild does in this case that worked of EL6.7 and didn't work on EL7.</p> <p>Thanks<br/> -Minh</p> <p>CentOS7 is the same as EL7 for our purposes. So yes, I did try it, and it works.</p> <p>I'm not sure which log you mean by "output log". But I'm also not seeing how that would help you since your build farm with lbuild is so incredibly different a normal build environment.</p> <p>Welcome to the club. We <em>all</em> wonder why lbuild works on one but not the other. In an lbuild-free environment, the builds appear to work on that platform.</p> <p>Regarding</p> <p>&gt;&gt; lbuild and the environment it operates in is sufficiently nonstandard for host and rpm builds that we'll have difficulty getting it to use any of the standard kernel module package methods without addressing some of it's deficiencies.</p> <p>We are using lbuild just as if anyone use rpmbuild to build the package. I believe we address one of the issue where kernel module package requires kernel-devel to install on the builders which we did.</p> <p>it would be great if we can understand more of the deficiencies that you mentioned. </p> <p>&gt;&gt;&gt; I'm not sure which log you mean by "output log". But I'm also not seeing how that would help you since your build farm with lbuild is so incredibly different a normal build environment.</p> <p>It would be nice if you can provide the commands and its' output that you built from EL7. I assume you'll run something like: autogen.sh, configure ...., make....</p> <p>Yes, I ran "autogen.sh &amp;&amp; ./configure &amp;&amp; make rpms".</p> <p>You have both successful and failed logs in lbuild paths for RHEL6.7 and RHEL7. Those are the logs and systems you need to be looking at in the short term.</p> <p>Yes, I am looking. I am mostly interested in rpmbuild output. BTW did you build locally with patched kernel?</p> <p>The build systems I am familiar with have a few key characteristics we are lacking for using this macro.</p> <ul class="alternate" type="square"> <li>Automatically installs packages to satisfy Requires: and BuildRequires: statements prior to invoking rpmbuild (or other native build mechanism).</li> </ul> <ul class="alternate" type="square"> <li>Builds only use libraries and objects which are installed in the build environment (host/guest/chroot) with the native package manager (rpm).</li> </ul> <p>Not all of the build systems do this, but ideally kernel module package builds also have:</p> <ul class="alternate" type="square"> <li>Insure that only a single instance of a package name.arch is installed. This is normally enforced for packages other than the kernel and related packages - but building kernel modules, it is important to be strict about kernel packages because of hazards in how the target kernel is determined, how kernel module dependencies are resolved, and how kernel source and header links are managed on some distributions.</li> </ul> <p>I moved forward a patch for kmod.<br/> <a href="http://review.whamcloud.com/#/c/12063/16" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12063/16</a> was build success for all kernels but there was .files generation from a spec file and rpms for rhel have ksym() instead of kernel(). The .files generation from spec was changed to rpm/*.files with lbuild modifying as suggested by Christopher, patchset 22 was fine except one build server RH7. At patchset 23 I`ve added Buildrequires: kernel-devel = %</p> {kversion} <p>and all redhat build failed as expected.<br/> To move forward this issue requires help from Intel Jenkins people, we need kernel-devel package with precise version to build valid kmod dependencies from kernel symbols.</p> <p>Peter Jones, could we get Intel to weigh in here? We are almost certainly going to need to get more attention from a person or persons at Intel to make any further progress.</p> <p>Alexander and Stephen have done some great work on this patch. The community has put in its effort to move Lustre leaps and bounds closer to proper packaging. Now we are all blocked by problems in Intel's build farm.</p> <p>We need to have a serious discussion about application of manpower and approach to resolving this problem. We need to make some firm plans now if we are going to get this done for 2.9.</p> <p>How do we move forward? Would a teleconference help?</p> <p>Hi Chris,</p> <p>I will starting looking into this and will gather more manpower and resources to figure out what issue with Intel's build farm and come up with a sensible solution.</p> <p>Great, Minh!</p> <p>started this by manually configure, make rpms but failed at make rpms</p> <p>make<span class="error">&#91;1&#93;</span>: Entering directory `/mnt/lustre-release'<br/> make<span class="error">&#91;1&#93;</span>: Leaving directory `/mnt/lustre-release'<br/> Installing lustre-2.7.64-1_g481176c.src.rpm<br/> error: Failed build dependencies:<br/> kernel-devel = 2.6.32-573.12.1.el6.x86_64 is needed by lustre-client-2.7.64-2.6.32_573.12.1.el6.x86_64_g481176c.x86_64<br/> make: *** <span class="error">&#91;rpms&#93;</span> Error 1<br/> <span class="error">&#91;root@onyx-24 lustre-release&#93;</span># rpm -qa | grep kernel-devel<br/> kernel-devel-2.6.32-573.12.1.el6.x86_64<br/> <span class="error">&#91;root@onyx-24 lustre-release&#93;</span># </p> <p>This is similar to what Intel jenkins reported. still looking, just FYI</p> <p>Minh, either use patch revision 22 for now or remove the "= %{kversion}" from the following line in the lustre.spec.in in patch revision 23:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>BuildRequires: kernel-devel = %{kversion}</pre> </div></div> <p>ah, I see, revision 22 still failed on el7 due to the extra x86_64 string in the patch name</p> <p>Minh, while that is a problem that needs resolution, the bigger issue that we need you focussed on is the kernel-devel package installation issue. For instance, while the Intel build farm says that the el6.7, inkernel, server build was successfully for patch revision 22, that isn't really correct. The build farm only checks that something came out of the build; it doesn't really check if what came out is correct.</p> <p>The kernel symbol requirements in the resulting binary rpms are incorrect because the correct kernel-devel package was not installed at packaging time.</p> <p>Chris, kernel-devel is installed whenever a new version released. I checked the builder node which built version 22 (onyx-8-sdf1-el6-x8664) and see kernel-devel installed</p> <p><span class="error">&#91;root@onyx-8-sdf1-el6-x8664 ~&#93;</span># rpm -qa | grep kernel-devel<br/> kernel-devel-2.6.32-573.12.1.el6.x86_64<br/> kernel-devel-2.6.32-573.3.1.el6.x86_64<br/> kernel-devel-2.6.32-573.7.1.el6.x86_64<br/> kernel-devel-2.6.32-573.8.1.el6.x86_64<br/> kernel-devel-2.6.32-504.12.2.el6.x86_64</p> <p>The next question that you need to ask yourself is this: "Are these the the kernel-devel packages for the correct kernel"?</p> <p>Keep in mind that Lustre is being built against a custom patched kernel. This kernel is built and packaged by lbuild during the lustre build process. You can see the packages for one distro here (this is linked off of the same patch revision 22, el6.7, inkernel, server build that I referenced ealier):</p> <p><a href="https://build.hpdd.intel.com/job/lustre-reviews/36780/arch=x86_64,build_type=server,distro=el6.7,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/36780/arch=x86_64,build_type=server,distro=el6.7,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/</a></p> <p>And here are the relevant rpms:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kernel-2.6.32-573.8.1.el6_lustre.g773a7e4.x86_64.rpm kernel-debuginfo-2.6.32-573.8.1.el6_lustre.g773a7e4.x86_64.rpm kernel-debuginfo-common-x86_64-2.6.32-573.8.1.el6_lustre.g773a7e4.x86_64.rpm kernel-devel-2.6.32-573.8.1.el6_lustre.g773a7e4.x86_64.rpm kernel-firmware-2.6.32-573.8.1.el6_lustre.g773a7e4.x86_64.rpm kernel-headers-2.6.32-573.8.1.el6_lustre.g773a7e4.x86_64.rpm </pre> </div></div> <p>You will note that the kernels that lbuild patches and builds have "_lustre.g773a7e4" appended to their revision number. The kernel packages that you listed do not contain that string, so I would expect that none of the kernel-devel packages that you listed are from the correct kernel.</p> <p>gotcha!</p> <p>Minh Diep, to check kmod dependencies you need to do</p> <p>rpm -qp --requires kmod-xxx.rpm | grep vfree<br/> the result should be kernel(vfree)<br/> for example</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rpm -qp --requires /lustre/lustre/kmod-lustre-client-tests-2.7.64-3.10.0_229.20.1.el7.x86_64_g773a7e4.x86_64.rpm | grep vfree kernel(vfree) = 0x999e8297 </pre> </div></div> <p>Patchset 22 have ksym(vfree) and it is wrong.</p> <p>Just a note: client builds are done using the distro kernel. So we need the distro -devel kernel packages for client builds and the _lustre.tag kernel packages for server builds.</p> <p>I noticed that spl and zfs also require kernel-devel*lustre*. are they not having the same problem?</p> <p>I`ve tried to check kmod-zfs, but fail. Download requires authorization. What is reason for separate access to lustre packages and kmod-zfs?</p> <p>Limitations imposed (or at least perceived to be imposed) by the CDDL vs GPL licensing incompatibility.</p> <p>Hi,</p> <p>With my latest revision built under <a href="https://build.hpdd.intel.com/job/lustre-reviews/37084/arch=x86_64,build_type=server,distro=el6.7,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/37084/arch=x86_64,build_type=server,distro=el6.7,ib_stack=inkernel/artifact/artifacts/RPMS/x86_64/</a></p> <p>the kmod dependencies on el6.7 seems to be correct</p> <p><span class="error">&#91;root@onyx-24 ~&#93;</span># rpm -qp --requires kmod-lustre-2.7.64-2.6.32_573.8.1.el6_lustre.g160868c.x86_64_g160868c.x86_64.rpm | grep vfree<br/> kernel(vfree) = 0x999e8297<br/> <span class="error">&#91;root@onyx-24 ~&#93;</span># </p> <p>I still need to figure the failure on el7 though</p> <p>Minh, that is fantastic! Did you do any spot checking to see if the packages work?</p> <p>Christopher J. Morrone (morrone2@llnl.gov) uploaded a new patch: <a href="http://review.whamcloud.com/18170" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18170</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> build: use %kernel_module_package in rpm spec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2ec18b2635b101e3d2661d49949ca68af170a1fc</p> <p>Chris, not yet. There will be more testing on this I am sure</p> <p>Any theories on that final<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/> issue with the RHEL7 build?</p> <p>We used to build el7 server on a shorter build directory (ie /var/lib/jenkins/tmp/el7_top_dir) because somehow rpmbuild doesn't work on long dir (ie /var/lib/jenkins/workspace/lustre-reviews/arch/x86_64/build_type/server/distro/el7/ib_stack/inkernel/BUILD/BUILD). I don't know why it failed with this kmod patch but not other. I changed it back to long dir to build in the same directory as git checkout; and this solved the issue.</p> <p>Installation test found some issues</p> <p>+ yum install -y kmod-lustre-osd-ldiskfs<br/> Loaded plugins: fastestmirror, security<br/> Setting up Install Process<br/> Determining fastest mirrors</p> <ul> <li>base: mirror.nexcess.net</li> <li>extras: mirror.compevo.com</li> <li>updates: mirror.chpc.utah.edu<br/> Resolving Dependencies<br/> --&gt; Running transaction check<br/> ---&gt; Package kmod-lustre-osd-ldiskfs.x86_64 0:2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116 will be installed<br/> --&gt; Finished Dependency Resolution</li> </ul> <p>Dependencies Resolved</p> <p>================================================================================<br/> Package Arch Version Repository Size<br/> ================================================================================<br/> Installing:<br/> kmod-lustre-osd-ldiskfs x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116<br/> lustre-build 2.8 M</p> <p>Transaction Summary<br/> ================================================================================<br/> Install 1 Package(s)</p> <p>Total download size: 2.8 M<br/> Installed size: 16 M<br/> Downloading Packages:<br/> Running rpm_check_debug<br/> Running Transaction Test<br/> Transaction Test Succeeded<br/> Running Transaction<br/> ^M Installing : kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre. 1/1<br/> find: `/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre-osd-ldiskfs': No such file or directory<br/> ^M Verifying : kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre. 1/1</p> <p>and interestingly, install lustre require to install lustre-dkms.</p> <p>Installing:<br/> lustre x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116 lustre-build 619 k<br/> Installing for dependencies:<br/> dkms noarch 2.2.0.3-30.git.7c3e7c5.el6 addon-epel6-x86_64 77 k<br/> libnvpair1 x86_64 0.6.4.2-1.el6 lustre-build 27 k<br/> libuutil1 x86_64 0.6.4.2-1.el6 lustre-build 32 k<br/> libzfs2 x86_64 0.6.4.2-1.el6 lustre-build 112 k<br/> libzpool2 x86_64 0.6.4.2-1.el6 lustre-build 389 k<br/> lustre-dkms noarch 2.7.65-1.el6 lustre-build 12 M<br/> lustre-osd-ldiskfs-mount x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116 lustre-build 13 k<br/> lustre-osd-zfs-mount x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116 lustre-build 7.4 k<br/> net-snmp-libs x86_64 1:5.5-54.el6_7.1 updates-centos6.7-x86_64 1.5 M<br/> spl-dkms noarch 0.6.4.2-1.el6 lustre-build 446 k<br/> zfs-dkms noarch 0.6.4.2-1.el6 lustre-build 1.9 M</p> <p>Transaction Summary<br/> ======================================================================================================================================================================================<br/> Install 12 Package(s)</p> <p>Just wanted to say I'm strongly supporting this feature. I very want something similar to Chris' patch being landed! I also think this is the good way to go. At CEA we are building our own kernel, (M)OFED and Lustre RPMS, in standard way, using standard tools. I would love building lustre with such patch.<br/> There is no reason this could not be possible in lbuild. </p> <p>Minh, can you please elaborate on the "interestingly, install lustre require to install lustre-dkms" statement? Are you saying that the "yum install -y kmod-lustre-osd-ldiskfs" command was the one that pulled in the lustre-dkms package?</p> <p>sorry for not being clear, it's yum -y lustre command that pulled in the lustre-dkms package</p> <p>so lustre requires lustre-osd</p> <p><span class="error">&#91;root@onyx-21vm4 ~&#93;</span># rpm -qp --requires ./lustre-2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_gb872116.x86_64.rpm | head<br/> rpmlib(VersionedDependencies) &lt;= 3.0.3-1<br/> kmod-lustre = 2.7.65<br/> lustre-osd <br/> lustre-osd-mount </p> <p>which lustre-dkms provides</p> <p><span class="error">&#91;root@onyx-21vm4 ~&#93;</span># rpm -qp --provides ./lustre-dkms-2.7.65-1.el6.noarch.rpm <br/> lustre-kmod = 2.7.65<br/> lustre-modules = 2.7.65<br/> lustre-osd <br/> lustre-osd-zfs = 2.7.65<br/> lustre-dkms = 2.7.65-1.el6<br/> <span class="error">&#91;root@onyx-21vm4 ~&#93;</span># </p> <p>Let's take stock of where we stand.</p> <p>Obviously, we need a rebase of the patch.</p> <p>Next we have the interaction between the dkms packages and the non-dkms packages. I think the solution there is probably pretty straight forward: don't put both packages in the same yum repository.</p> <p>What else do we need to work through?</p> <p>I have updated the patch with more dependencies, please review. This resolved the conflict with dkms package.<br/> We will need to automate this for testing. this is is progress now.</p> <p>Could you explain, why the next line was added?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Requires: %{requires_kmod_name} = %{requires_kmod_version}\n\ </pre> </div></div> <p>kmod adds requires on the symbols, so old style package references looks strange. </p> <p>you're right, we don't need that.</p> <p>I found that osd_ldiskfs.ko is not in weak-updates/... directory</p> <p><span class="error">&#91;root@onyx-21vm4 ~&#93;</span># uname -a<br/> Linux onyx-21vm4.onyx.hpdd.intel.com 2.6.32-573.18.1.el6.x86_64 #1 SMP Tue Feb 9 22:46:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux<br/> <span class="error">&#91;root@onyx-21vm4 ~&#93;</span># ls /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/*ldiskfs.ko<br/> /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/ldiskfs.ko<br/> /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko<br/> <span class="error">&#91;root@onyx-21vm4 ~&#93;</span># ls /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/lustre/fs/*ldiskfs.ko<br/> /lib/modules/2.6.32-573.18.1.el6.x86_64/weak-updates/lustre/fs/ldiskfs.ko<br/> <span class="error">&#91;root@onyx-21vm4 ~&#93;</span># </p> <p>Is this expected or this is not working on lustre patched server?</p> <p>There are some differences for a rpms between SLES and RHEL at the rpm kmod scripts.<br/> RHEL</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rpm --scripts -qp kmod-lustre-osd-ldiskfs-2.8.50-2.6.32_573.12.1.el6_lustre.g33b2752.x86_64_gd0671d9.x86_64.rpm postinstall scriptlet (using /bin/sh): if [ -e "/boot/System.map-2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" ]; then /sbin/depmod -aeF "/boot/System.map-2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" "2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" &gt; /dev/null || : fi modules=( $(find /lib/modules/2.6.32-573.12.1.el6_lustre.g33b2752.x86_64/extra/lustre-osd-ldiskfs | grep '\.ko$') ) if [ -x "/sbin/weak-modules" ]; then printf '%s\n' "${modules[@]}" | /sbin/weak-modules --add-modules fi preuninstall scriptlet (using /bin/sh): rpm -ql kmod-lustre-osd-ldiskfs-2.8.50-2.6.32_573.12.1.el6_lustre.g33b2752.x86_64_gd0671d9.x86_64 | grep '\.ko$' &gt; /var/run/rpm-kmod-lustre-osd-ldiskfs-modules postuninstall scriptlet (using /bin/sh): if [ -e "/boot/System.map-2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" ]; then /sbin/depmod -aeF "/boot/System.map-2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" "2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" &gt; /dev/null || : fi modules=( $(cat /var/run/rpm-kmod-lustre-osd-ldiskfs-modules) ) rm /var/run/rpm-kmod-lustre-osd-ldiskfs-modules if [ -x "/sbin/weak-modules" ]; then printf '%s\n' "${modules[@]}" | /sbin/weak-modules --remove-modules fi </pre> </div></div> <p>SLES</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> rpm --scripts -qp lustre-osd-ldiskfs-kmp-default-2.8.50_3.0.101_0.47.71-3.0.101_0.47.71_lustre.g33b2752_default_gd0671d9.x86_64.rpm postinstall scriptlet (using /bin/sh): nvr=lustre-osd-ldiskfs-kmp-default-2.8.50_3.0.101_0.47.71-3.0.101_0.47.71_lustre.g33b2752_default_gd0671d9 wm2=/usr/lib/module-init-tools/weak-modules2 if [ -x $wm2 ]; then /bin/bash -${-/e/} $wm2 --add-kmp $nvr fi preuninstall scriptlet (using /bin/sh): nvr=lustre-osd-ldiskfs-kmp-default-2.8.50_3.0.101_0.47.71-3.0.101_0.47.71_lustre.g33b2752_default_gd0671d9 rpm -ql $nvr | sed -n '/\.ko$/p' &gt; /var/run/rpm-$nvr-modules postuninstall scriptlet (using /bin/sh): nvr=lustre-osd-ldiskfs-kmp-default-2.8.50_3.0.101_0.47.71-3.0.101_0.47.71_lustre.g33b2752_default_gd0671d9 modules=( $(cat /var/run/rpm-$nvr-modules) ) rm -f /var/run/rpm-$nvr-modules if [ ${#modules[*]} = 0 ]; then echo "WARNING: $nvr does not contain any kernel modules" &gt;&amp;2 exit 0 fi wm2=/usr/lib/module-init-tools/weak-modules2 if [ -x $wm2 ]; then printf '%s\n' "${modules[@]}" | /bin/bash -${-/e/} $wm2 --remove-kmp $nvr fi </pre> </div></div> <p>The main point is RHEL uses modules directory equal to rpm package name to find and add *ko, for example <tt>extra/lustre-osd-ldiskfs</tt> . And SLES uses rpm package to find and add modlues. We have a different package name lustre-osd-ldiskfs lustre-tests etc., but use one directory to install all of them extra/lustre/fs/ and this could bring RHEL to some kind of a problems.<br/> We installed rpms and updated a kernel for a client. The Lustre works fine after update. I suppose that kmod-lustre do postintall weak -add for all *ko from extra/lustre/ and we didn`t have a problems that I`ve mentioned above.</p> <p>Now with your question,<br/> Did you see any error when you updated a kernel, it may have another version of kernel symbol and weak-update should fail, especially for ldiskfs*.</p> <p>FYI, after expanding %kernel_module_package, the correct module installation directory can be obtained via %kernel_module_package_moddir instead of the inline selection and assignment to %kmoddir introduced with <a href="https://jira.whamcloud.com/browse/LU-4124" title="Make module installation directory flexible" class="issue-link" data-issue-key="LU-4124"><del>LU-4124</del></a>.</p> <p>As Alexander mentioned, I suspect a kernel symbol mismatch for osd_ldiskfs trying to install for a non-lustre kernel.</p> <p>Did you see any error when you updated a kernel, it may have another version of kernel symbol and weak-update should fail, especially for ldiskfs*.</p> <p>&gt;&gt; No, I did not. However, we can't mount lustre dev because of missing osd-ldiskfs module (after upgrade)</p> <p>perhaps I did, here is the command</p> <p>yum install -y kernel-2.6.32-573.12.1.el6_lustre.gb872116.x86_64 kmod-lustre kmod-lustre-osd-ldiskfs</p> <p>Loaded plugins: fastestmirror, security<br/> Setting up Install Process<br/> Resolving Dependencies<br/> --&gt; Running transaction check<br/> ---&gt; Package kernel.x86_64 0:2.6.32-573.12.1.el6_lustre.gb872116 will be installed<br/> ---&gt; Package kmod-lustre.x86_64 0:2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b will be installed<br/> ---&gt; Package kmod-lustre-osd-ldiskfs.x86_64 0:2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b will be installed<br/> --&gt; Processing Dependency: lustre-osd-ldiskfs-mount = 2.7.65 for package: kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b.x86_64<br/> --&gt; Running transaction check<br/> ---&gt; Package lustre-osd-ldiskfs-mount.x86_64 0:2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b will be installed<br/> --&gt; Finished Dependency Resolution</p> <p>Dependencies Resolved</p> <p>================================================================================<br/> Package Arch Version Repository Size<br/> ================================================================================<br/> Installing:<br/> kernel x86_64 2.6.32-573.12.1.el6_lustre.gb872116<br/> lustre-build 30 M<br/> kmod-lustre x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b<br/> lustre-build 26 M<br/> kmod-lustre-osd-ldiskfs x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b<br/> lustre-build 2.8 M<br/> Installing for dependencies:<br/> lustre-osd-ldiskfs-mount x86_64 2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86_64_g0283f6b<br/> lustre-build 13 k</p> <p>Transaction Summary<br/> ================================================================================<br/> Install 4 Package(s)</p> <p>Total download size: 59 M<br/> Installed size: 298 M<br/> Downloading Packages:<br/> --------------------------------------------------------------------------------<br/> Total 55 MB/s | 59 MB 00:01<br/> Running rpm_check_debug<br/> Running Transaction Test<br/> Transaction Test Succeeded<br/> Running Transaction<br/> ^M Installing : kernel-2.6.32-573.12.1.el6_lustre.gb872116.x86_64 1/4<br/> ^M Installing : lustre-osd-ldiskfs-mount-2.7.65-2.6.32_573.12.1.el6_lustre 2/4<br/> ^M Installing : kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre. 3/4<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_unregister_type<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_manual_cleanup<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_kbytestotal_seq_show<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_buf_realloc<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lustre_lma_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol seq_client_fini<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_register_type<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_fail_loc<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_oh_tally_log2<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_env_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lbug_with_loc<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_restore_sigs<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_get<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_log_goto<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_debug_msg<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol __cfs_fail_check_set<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_uuid_unparse<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_filesfree_seq_show<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_remove<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_register_stats<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_counter_add<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_device_put<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_conn2export<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol seq_server_alloc_meta<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol linkea_entry_unpack<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_purge<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol fld_local_lookup<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol server_name2index<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_env_fini<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_buf_free<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_op_begin<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol tgt_server_data_update<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_clear_sigpending<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol obd_memory<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_start<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_process_proc_param<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_dev_del_linkage<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lustre_lma_swab<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_register<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_hash_is_empty<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_txn_hook_start<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_debug_dumpstack<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_oh_sum<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol eq_client_alloc_fid<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_connect<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_fini<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_log_return<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_block_sigsinv<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_alloc_stats<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_txn_hook_stop<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_cdebug_printer<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_filestotal_seq_show</p> <p>WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_cdebug_printer<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_filestotal_seq_show<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_degister_many<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_otable_features<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_device_get<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_object_fini<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_device_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_object_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_register_many<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol statfs_pack<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_fini<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_prepare<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_init_finish<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_kbytesfree_seq_show<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_free_stats<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_revive_many<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_enter<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_kbytesavail_seq_show<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_single_release<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_counter_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_env_refill<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol linkea_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_oh_tally<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_object_put<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol pop_ctxt<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_kmem_fini<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_quota_glb_features<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_version_get<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol LU_DOT_LUSTRE_FID<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_acct_features<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_write_u64_helper<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_site_print<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_locate_at<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_directory_features<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_kmem_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_subsystem_debug<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol seq_client_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_search_type<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_init<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_write_helper<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_seq_create<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_lookup_dir<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_counter_sub<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_exit<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_debug</p> <p>WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_counter_sub<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_exit<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol libcfs_debug<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_oh_clear<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_device_fini<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_op_end<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_fini<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lu_context_key_quiesce_many<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol lprocfs_dt_blksize_seq_show<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol qsd_op_adjust<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol push_ctxt<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol dt_txn_hook_commit<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol cfs_fail_val<br/> WARNING: /lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre/fs/osd_ldiskfs.ko needs unknown symbol class_disconnect<br/> find: `/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre-osd-ldiskfs': No such file or directory<br/> ^M Installing : kmod-lustre-2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86 4/4<br/> ^M Verifying : kmod-lustre-osd-ldiskfs-2.7.65-2.6.32_573.12.1.el6_lustre. 1/4<br/> ^M Verifying : kernel-2.6.32-573.12.1.el6_lustre.gb872116.x86_64 2/4<br/> ^M Verifying : kmod-lustre-2.7.65-2.6.32_573.12.1.el6_lustre.gb872116.x86 3/4<br/> ^M Verifying : lustre-osd-ldiskfs-mount-2.7.65-2.6.32_573.12.1.el6_lustre 4/4</p> <p>Since you are building the modules with a patched kernel, it is not surprising that it is unable to resolve symbols when it attempts weak-updates for the unpatched kernel. Anytime you change the KABI without changing the flavor, this is the result. What is surprising is that only osd_ldiskfs.ko has trouble - this demonstrates great progress removing dependencies on the patched kernel!</p> <p>To test weak updates on servers, try installing the server modules on a system with a different (and later) version #. Ie, build modules for 2.6.32-573.12.1.el6_lustre.foo and install them on 2.6.32-573.18.1.el6_lustre.fop</p> <p>This is curious:<br/> find: `/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre-osd-ldiskfs': No such file or directory</p> <p>Ah, I was hoping that this ticket let us use weak updates on unpatched kernel. I will try your suggestion.</p> <p>the line above is coming from the post install script</p> <p>rpm --scripts -qp kmod-lustre-osd-ldiskfs-2.8.50-2.6.32_573.12.1.el6_lustre.g33b2752.x86_64_gd0671d9.x86_64.rpm <br/> postinstall scriptlet (using /bin/sh):<br/> if [ -e "/boot/System.map-2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" ]; then<br/> /sbin/depmod -aeF "/boot/System.map-2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" "2.6.32-573.12.1.el6_lustre.g33b2752.x86_64" &gt; /dev/null || :<br/> fi</p> <p>modules=( $(find /lib/modules/2.6.32-573.12.1.el6_lustre.g33b2752.x86_64/extra/lustre-osd-ldiskfs | grep '\.ko$') )<br/> if [ -x "/sbin/weak-modules" ]; then<br/> printf '%s\n' "${modules<span class="error">&#91;@&#93;</span>}" | /sbin/weak-modules --add-modules<br/> fi</p> <p>the path should be /lib/modules/2.6.32-573.12.1.el6_lustre.g33b2752.x86_64/extra/lustre/fs</p> <p>The patch does allow using weak modules with unpatched kernels. This patch doesn't care whether the kernel is patched or not; it works with both. However not all kernels offer identical symbols, so one build of lustre will not magically work with all kernels.</p> <p>When <a href="https://jira.whamcloud.com/browse/LU-684" title="replace dev_rdonly kernel patch with dm-flakey" class="issue-link" data-issue-key="LU-684"><del>LU-684</del></a> is done, I believe that we will at last have eliminated the kernel symbol changes that Lustre is currently introducing in the Intel build farm.</p> <p>Minh, are you working on splitting apart the yum repos for the dkms and standard lustre builds?</p> <p>What else is currently on our list of blockers to getting this ticket done? Anything we can help with?</p> <p>I am working on moving the inline Require in to a file. will upload a version shortly. I have verified that we do not need the split the repo.</p> <p>This is one perhaps minor issue notice above about this error</p> <p>find: `/lib/modules/2.6.32-573.12.1.el6_lustre.gb872116.x86_64/extra/lustre-osd-ldiskfs': No such file or directory</p> <p>when installing kmod-lustre-osd-ldiskfs. If you can help with this, this would be great.</p> <p>and when I installed: yum install kmod-lustre-osd-zfs</p> <p>I see error like these</p> <p>--&gt; Processing Dependency: ksym(arc_add_prune_callback) = 0x5bd5668d for package: kmod-lustre-osd-zfs-2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852.x86_64<br/> --&gt; Processing Dependency: ksym(__cv_broadcast) = 0x1d5ed383 for package: kmod-lustre-osd-zfs-2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852.x86_64<br/> ---&gt; Package lustre-osd-zfs-mount.x86_64 0:2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852 will be installed<br/> --&gt; Finished Dependency Resolution<br/> Error: Package: kmod-lustre-osd-zfs-2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852.x86_64 (lustre-build)<br/> Requires: ksym(zap_remove) = 0xce8deead<br/> Error: Package: kmod-lustre-osd-zfs-2.8.50-2.6.32_431.29.2.el6_lustre.g57d7852.x86_64_g57d7852.x86_64 (lustre-build)<br/> Requires: ksym(sa_buf_rele) = 0xfb208743</p> <p>Is ZFS installed?</p> <p>yes, I installed zfs before that and verified that zfs, spl, kmod-zfs, kmod-spl were on the system</p> <p>You are hacking some internal script to get it to make the kernel's symbol dependencies be named "ksym" instead of "kernel", right? Did that hack go too far and change the name for symbols that are in external modules as well?</p> <p>I don't think so, all we do is edit the scripts to use the build path, not /usr/src/...</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> cp $RPM_HELPERS_DIR/{symset-table,find-requires{,.ksyms}} . FIND_REQUIRES=<span class="code-quote">"$(pwd)/find-requires"</span> chmod 755 {symset-table,find-requires{,.ksyms}} local tmp=<span class="code-quote">"$(pwd)"</span> tmp=<span class="code-quote">"${tmp<span class="code-comment">//\//\\/}"</span> </span> ed find-requires &lt;&lt;EOF 1a set -x . /|.*find-requires.ksyms/s/|/| bash -x/ g/ [^ ]*\/\(find-requires\.ksyms\)/s<span class="code-comment">// $tmp\/\1/g </span>wq EOF ed find-requires.ksyms &lt;&lt;EOF 1a set -x . g/\/.*\/\(symset-table\)/s<span class="code-comment">//$tmp\/\1/g </span>g/\(\/usr\/src\/kernels\/\)/s<span class="code-comment">//$tmp\/reused\1/g </span>wq EOF ed symset-table &lt;&lt;EOF 1a set -x . g/\(\/boot\/\)/s<span class="code-comment">//$tmp\/reused\1/g </span>g/\(\/usr\/src\/kernels\/\)/s<span class="code-comment">//$tmp\/reused\1/g </span>wq EOF </pre> </div></div> <p>I saw a refresh of this patch. Does this mean the issues of the Intel build farm have been resolved?</p> <p>There were just a large number of conflicts piling up so I did a refresh. No other changes, so any outstanding problems are probably still there.</p> <p>somehow build 38351 did not produce any kmod-lustre rpm</p> <p>Could you be more specific? It looks to like the the rhel7 builder for build 38351 produced kmode-lustre rpms:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/lustre-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/kmod-lustre-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/kmod-lustre-osd-ldiskfs-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/lustre-osd-ldiskfs-mount-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/kmod-lustre-osd-zfs-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/lustre-osd-zfs-mount-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/lustre-source-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/lustre-tests-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/kmod-lustre-tests-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/lustre-iokit-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm Wrote: /tmp/rpmbuild-lustre-jenkins-xBBcs7qm/RPMS/x86_64/lustre-debuginfo-2.8.51_36_g4782472-3.10.0_327.13.1.el7_lustre.x86_64.x86_64.rpm </pre> </div></div> <p>Oh, they were produced, but it looks like lbuild failed to copy them? Hmm, did I drop some change from lbuild in the rebase? I'll look into that.</p> <p>Oh, whew, I didn't change anything by accident. But the "<a href="https://jira.whamcloud.com/browse/LU-7887" title="lbuild not building lnetctl" class="issue-link" data-issue-key="LU-7887"><del>LU-7887</del></a> lbuild: use "make rpms" for building Lustre RPM" made an unfortunate lbuild change that we now have to deal with. I'll refresh the patch some time today to address that.</p> <p>It looks like patch set 30 dealt with the <a href="https://jira.whamcloud.com/browse/LU-7887" title="lbuild not building lnetctl" class="issue-link" data-issue-key="LU-7887"><del>LU-7887</del></a> change just fine.</p> <p>But testing is all failing.</p> <p>Minh, it looks like there are problems at package installation time. In one of the logs named "node-provisioning-1.node-provisioning_1.autotest.trevis-38vm1.log" I am seeing the following failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>03:36:43:Attempt 3 03:36:44:yum install output: Loaded plugins: fastestmirror, security Setting up Install Process Loading mirror speeds from cached hostfile No package lustre-osd-ldiskfs available. Error: Nothing to do 03:36:44:Yum install was unsuccessful. 03:36:44:Exhausted installation attempts of package lustre-osd-ldiskfs. 03:36:44:File system install complete. 03:36:44:after_prepare_nodes complete for trevis-38vm3 03:36:44:Preparing nodes complete! </pre> </div></div> <p>First of all, the scripts are failing to detect the failure of the installation process. The provisioning step is declared a success even though it it failing.</p> <p>Next, the scripts appear to be trying to install lustre-osd-ldiskfs which is not a valid package name under this patch. We probably need to update the build farm scripts to understand the new package names.</p> <p>got it. I'll check it out</p> <p>actually the error was<br/> --&gt; Finished Dependency Resolution<br/> Error: Package: kmod-lustre-osd-ldiskfs-2.8.52_37_g9d5323c-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)<br/> Requires: ksym(kill_block_super) = 0x0f72d0ce<br/> Error: Package: kmod-lustre-osd-ldiskfs-2.8.52_37_g9d5323c-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)<br/> Requires: ksym(jbd2_journal_destroy) = 0xb5132dfa</p> <p>we need to rebase due to a bug in <a href="https://jira.whamcloud.com/browse/LU-7887" title="lbuild not building lnetctl" class="issue-link" data-issue-key="LU-7887"><del>LU-7887</del></a></p> <p>with the latest build yum install kmod-lustre-osd-zfs still failed dependencies. I have install zfs, kmod-zfs, spl, ...</p> <p>---&gt; Package lustre-osd-zfs-mount.x86_64 0:2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64 will be installed<br/> --&gt; Finished Dependency Resolution<br/> Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)<br/> Requires: ksym(dmu_objset_disown) = 0xde76f7a7<br/> Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)<br/> Requires: ksym(sa_lookup) = 0x9fec99c4<br/> Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)<br/> Requires: ksym(spa_writeable) = 0xd3eb11e1<br/> Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)<br/> Requires: ksym(zap_add_uint64) = 0xfe7bbafd<br/> Error: Package: kmod-lustre-osd-zfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64 (lustre-build)</p> <p>here is what I have on the node</p> <p><span class="error">&#91;root@trevis-28vm2 yum.repos.d&#93;</span># rpm -qa | grep lustre<br/> lustre-osd-ldiskfs-mount-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64<br/> lustre-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64<br/> kernel-headers-2.6.32-573.22.1.el6_lustre.x86_64<br/> kmod-lustre-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64<br/> kmod-lustre-osd-ldiskfs-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64<br/> lustre-tests-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64<br/> kmod-spl-2.6.32-573.22.1.el6_lustre.x86_64-0.6.4.2-1.el6.x86_64<br/> kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64-0.6.4.2-1.el6.x86_64<br/> kernel-2.6.32-573.22.1.el6_lustre.x86_64<br/> kernel-devel-2.6.32-573.22.1.el6_lustre.x86_64<br/> lustre-iokit-2.8.52_45_gbef8586-2.6.32_573.22.1.el6_lustre.x86_64.x86_64<br/> kernel-firmware-2.6.32-573.22.1.el6_lustre.x86_64<br/> <span class="error">&#91;root@trevis-28vm2 yum.repos.d&#93;</span># </p> <p>OK, so find the rpm for the kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64-0.6.4.2-1.el6.x86_64 packge. Run this command on it:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rpm -qp --provides &lt;rpm file name&gt; | grep dmu_objset_disown</pre> </div></div> <p>What symbol version does it say that dmu_objest_disown has in that package?</p> <p>If it is anything other than "0xde76f7a7", that will explain why you are getting this error. If the line matches, then we'll move on to other problems.</p> <p>right, that's the issue. kmod-zfs doesn't provide any symbol.</p> <ol> <li>rpm -qp --provides ./kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64-0.6.4.2-1.el6.x86_64.rpm<br/> kernel-modules-for-kernel = 2.6.32-573.22.1.el6_lustre.x86_64<br/> kmod-zfs-uname-r = 2.6.32-573.22.1.el6_lustre.x86_64<br/> zfs-kmod = 0.6.4.2-1.el6<br/> kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64 = 0.6.4.2-1.el6<br/> kmod-zfs-2.6.32-573.22.1.el6_lustre.x86_64(x86-64) = 0.6.4.2-1.el6</li> </ol> <p>Great! That seems like the next Intel buildfarm problem that you need to tackle.</p> <p>Let me know if I can help in any way.</p> <p>I look at <a href="http://zfsonlinux.org/generic-rpm.html" class="external-link" target="_blank" rel="nofollow noopener">http://zfsonlinux.org/generic-rpm.html</a> and notice the steps to build zfs mod as below</p> <p>$ cd spl-x.y.z<br/> $ ./configure<br/> $ make rpm-utils rpm-kmod</p> <ol> <li>Install the spl packages, they are required to build zfs.<br/> $ sudo yum localinstall \<br/> kmod-spl-devel-x.y.z-r.dist.arch.rpm \<br/> kmod-spl-devel-kernel-x.y.z-r.dist.arch.rpm \<br/> kmod-spl-kernel-x.y.z-r.dist.arch.rpm \<br/> spl-x.y.z-r.dist.arch.rpm</li> </ol> <p>$ cd ../zfs-x.y.z<br/> $ ./configure<br/> $ make rpm-utils rpm-mod</p> <p>and we are using rpm build. I'll start there</p> <p>Chris, I have repeated the build manually, (ie not using lbuild) and it still failed the dependency. I don't think this is lbuild issue but building kmod-lustre-osd-zfs with kmod-zfs issue</p> <p>You are saying that the kmod-zfs package has no ksym symbols under the Provides even when you build it yourself?</p> <p>Alright, can you step me through the details of how you built zfs manually?</p> <p>here are the steps</p> <p> wget <a href="http://archive.zfsonlinux.org/downloads/zfsonlinux/spl/spl-0.6.4.2.tar.gz" class="external-link" target="_blank" rel="nofollow noopener">http://archive.zfsonlinux.org/downloads/zfsonlinux/spl/spl-0.6.4.2.tar.gz</a><br/> wget <a href="http://archive.zfsonlinux.org/downloads/zfsonlinux/zfs/zfs-0.6.4.2.tar.gz" class="external-link" target="_blank" rel="nofollow noopener">http://archive.zfsonlinux.org/downloads/zfsonlinux/zfs/zfs-0.6.4.2.tar.gz</a></p> <p>tar zxvf spl-0.6.4.2.tar.gz <br/> cd spl-0.6.4.2</p> <p> ./autogen.sh <br/> ./configure<br/> make rpm-utils rpm-kmod</p> <p>install all spl rpm</p> <p>tar zxvf zfs-0.6.4.2.tar.gz <br/> cd zfs-0.6.4.2</p> <p> ./autogen.sh <br/> ./configure<br/> make rpm-utils rpm-kmod</p> <p>then clone and checkout the patch<br/> sh autogen.sh <br/> ./configure --with-zfs=/usr/src/zfs-0.6.4.2/ --with-spl=/usr/src/spl-0.6.4.2/ --without-ldiskfs<br/> make rpm</p> <p>What distro is that on? What kernel version?</p> <p>What is the output of "rpm -qp --provides" for the kmod-zfs package that results?</p> <p><span class="error">&#91;root@onyx-21vm4 lustre-release&#93;</span># uname -a<br/> Linux onyx-21vm4.onyx.hpdd.intel.com 2.6.32-573.22.1.el6.x86_64 #1 SMP Wed Mar 23 03:35:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux<br/> <span class="error">&#91;root@onyx-21vm4 lustre-release&#93;</span># </p> <p><span class="error">&#91;root@onyx-21vm4 zfs-0.6.4.2&#93;</span># rpm -qp --provides kmod-zfs-2.6.32-573.22.1.el6.x86_64-0.6.4.2-1.el6.x86_64.rpm <br/> kernel-modules-for-kernel = 2.6.32-573.22.1.el6.x86_64<br/> kmod-zfs-uname-r = 2.6.32-573.22.1.el6.x86_64<br/> zfs-kmod = 0.6.4.2-1.el6<br/> kmod-zfs-2.6.32-573.22.1.el6.x86_64 = 0.6.4.2-1.el6<br/> kmod-zfs-2.6.32-573.22.1.el6.x86_64(x86-64) = 0.6.4.2-1.el6<br/> <span class="error">&#91;root@onyx-21vm4 zfs-0.6.4.2&#93;</span># </p> <p>Minh, you are going to want to add the configure option "--with-spec=redhat" when building both spl and zfs for the centos/rhel systems. That will get you the packages with the needed symbol dependencies.</p> <p>During my investigation of the previous package installation problem, I found another problem. Take a look at the postinstall script from the kmod-lustre-osd-zfs package generated under RHEL6.7:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>postinstall scriptlet (using /bin/sh): if [ -e "/boot/System.map-2.6.32-573.26.1.el6.x86_64" ]; then /sbin/depmod -aeF "/boot/System.map-2.6.32-573.26.1.el6.x86_64" "2.6.32-573.26.1.el6.x86_64" &gt; /dev/null || : fi modules=( $(find /lib/modules/2.6.32-573.26.1.el6.x86_64/extra/lustre-osd-zfs | grep '\.ko$') ) if [ -x "/sbin/weak-modules" ]; then printf '%s\n' "${modules[@]}" | /sbin/weak-modules --add-modules fi </pre> </div></div> <p>Note the path for the find command. It is looking in /lib/modules/2.6.32-573.26.1.el6.x86_64/extra/lustre-osd-zfs for the module, but that directory doesn't exist.</p> <p>I think that the simplest solution would probably just to make those subdirectories and move the modules in the %install section of the spec file. I'll give that a try.</p> <p>FYI, I am about to push another update to the patch adding a couple of more Obsoletes entries.</p> <p>Chris, have you (or anyone) tried to install kmod-lustre-osd-zfs from your own build?</p> <p>OK, patch set 38 addresses every issue that I know about in the patch. I tested it through package installation on a RHEL6.7 VM, and all went well. The dependencies on the zfs package all matched up, and the symlinks from all of the modules looked like they were created correctly, even for the osd modules (that wasn't working on RHEL until one of my updates last week).</p> <p>This <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> patch together with the patch from <a href="https://jira.whamcloud.com/browse/LU-7643" title="Remove kernel version string from Lustre release field" class="issue-link" data-issue-key="LU-7643"><del>LU-7643</del></a> are already allowing Lustre to build cleanly under LLNL's local koji build farm. That is a really nice improvement.</p> <p>The only problem that I know of now is that we need to get a version of ZFS built in Intel's build far that lists all of the provided symbols in the package Provides: section. We know one way to fix that (add --with-spec=redhat to the configure line on RHEL-like systems).</p> <p>Barring undiscovered issues, I think the patch is pretty darn close to complete.</p> <p>that's great news!<br/> I am curious as to why zfs configure script does not detect RHEL-like systems but have to manually specified by users</p> <p>Chris, I noticed when we use 'generic' spec, we produced two kmod-spl-devel</p> <p>kmod-spl-devel-0.6.4.2-1.el6.x86_64.rpm<br/> kmod-spl-devel-2.6.32-573.22.1.el6.x86_64-0.6.4.2-1.el6.x86_64.rpm</p> <p>but on 'redhat' spec, we only have one</p> <p>kmod-spl-devel-0.6.4.2-1.el6.x86_64.rpm</p> <p>I don't think there is anything to be concerned about there.</p> <p>is kmod-spl-devel-0.6.4.2-1.el6.x86_64.rpm the same between the 'generic' and 'redhat' spec?</p> <p>You can always rpm -qpl the package to find out. My guess is that there are differences in the contents. Why is this an issue?</p> <p>I am still looking into this. the 'redhat' spec is building against the builder's kernel, not the one we want for lustre.</p> <p>Chris, it seems to me that redhat/*-kmod.spec does not allow to build any kernel other than the one in /usr/src</p> <p>%define ksrc %{_usrsrc}/kernels/%</p> {kverrel} <p>in order to use it for lustre, we need to install lustre kernel onto the builder after we patched the kernel.</p> <p>any advice?</p> <blockquote> <p>%define ksrc %{_usrsrc}/kernels/%</p></blockquote> <p>You should be able to override the %{_usrsrc}} variable (either at the build invocation time, or in <tt>~/.rpmmacros</tt>) value to point at where you unpacked your kernel no?</p> <p>Yeah, getting the Intel buildfarm to do things normally (actually install the packages that are BuildRequires of lustre) is probably not going to happen any time soon. So in the mean time we are probably stuck with various variable overrides.</p> <p>I don't think that overriding _usrsrc will work on its own because the zfs "redhat" zfs-kmod.spec file uses %kernel_module_package, and that will always look instead for a properly installed kernel. That can be overriden, but that will take modifications to the zfs-kmod.spec script. I am none too certain that the zfs community is going to accept patches that dirty up their spec file for a completely non standard build environment.</p> <p>So maybe we should go back to the "generic" zfs-kmod.spec file. The "generic" spec file is kmods 2 based and designed to work well in build environments like rpmfusion. It sounds like that spec file was easier to build in the Intel buildfarm. The problem with those packages is that they were failing to list Provides for the provided kernel symbols. I verified that issue on a clean RHEL6.7 image. The zfs community is going to be alot more receptive to allowing a change to add those Provides.</p> <p>So I think we should put some effort into figuring out how to add the kernel symbol Provides to the <em>generic</em> zfs spec files.</p> <p>yes, I am able to build spl with spl-kmod.spec modified; only a few lines like below</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">31,34d30 &lt; %<span class="code-keyword">if</span> !%{defined kernels} &lt; %define kernels %{kverrel} &lt; %endif &lt; 37d32 &lt; %<span class="code-keyword">if</span> !%{defined ksrc} 39d33 &lt; %endif 51d44 &lt; 104c97 &lt; %{__rm} -f %{buildroot}/lib/modules/%{kernels}/modules.* --- &gt; %{__rm} -f %{buildroot}/lib/modules/%{kverrel}/modules.* 106c99 &lt; %{__chmod} u+x %{buildroot}/lib/modules/%{kernels}/extra<span class="code-comment">/*/*/</span>* --- &gt; %{__chmod} u+x %{buildroot}/lib/modules/%{kverrel}/extra<span class="code-comment">/*/*/</span>* 112d104 &lt; /lib/modules/%{kernels}/extra<span class="code-comment">/*/*/</span>* </pre> </div></div> <p>If you think zfs community won't allow such change for flexibility, then I will pursue the 'generic' spec</p> <p>Just a quick tip: always use unified diff format: "diff -u". A decent set of standard options would be "diff -uNp". Better yet, use git. The "git diff" command does the right thing by default. </p> <p>update: latest version with --define "_use_internal_dependency_generator 0" added worked!!<br/> I have verified osd-zfs loaded with kmod-zfs and lustre mounted on the server. client can move to later kernel without problem.<br/> I am proceeding to investigate why maloo failed</p> <p>Chris, I want to review the packages and provides; just to make sure</p> <p><span class="error">&#91;root@onyx-21vm7 build_type-server_distro-el6.7_arch-x86_64_ib_stack-inkernel&#93;</span># rpm -qp --provides lustre-dkms-2.8.52_70_g3660338-1.el6.noarch.rpm <br/> lustre-kmod = 2.8.52_70_g3660338<br/> lustre-modules = 2.8.52_70_g3660338<br/> lustre-osd <br/> lustre-osd-zfs = 2.8.52_70_g3660338<br/> lustre-dkms = 2.8.52_70_g3660338-1.el6</p> <p>should lustre-dkms also provides lustre-osd-ldiskfs?</p> <p>I have never reviewed what they did in the dkms packaging.</p> <p>I am investigating an sles12sp1 build failure that could be from this patch</p> <p><a href="https://build.hpdd.intel.com/job/lustre-reviews/39104/arch=x86_64,build_type=server,distro=sles12,ib_stack=inkernel/consoleText" class="external-link" target="_blank" rel="nofollow noopener">https://build.hpdd.intel.com/job/lustre-reviews/39104/arch=x86_64,build_type=server,distro=sles12,ib_stack=inkernel/consoleText</a></p> <p>d4d54bfba3e5f20ab369aa69b0f9b6dcfab1cdce debuginfo(build-id) = d8c98b864c6e0ac2360a32da503bd15d7cbbc263 debuginfo(build-id) = da62063a63187395fb3c9b3aae6e3541ed005684 debuginfo(build-id) = da697fd1e5187bf2f6ddc0dc1528c9050b51f08d debuginfo(build-id) = e46c8a226902868165db3894ee0450d05be00325 debuginfo(build-id) = f6cfd424f753677f8f61b05c6326af88cdafc5da debuginfo(build-id) = f8134d77ab4d46f6c4cd994efab8518e9e8b8372 lustre-tests-debuginfo = 2.8.53_27_ga8b7e8c-3.12.57_60.35_lustre_default lustre-tests-debuginfo(x86-64) = 2.8.53_27_ga8b7e8c-3.12.57_60.35_lustre_default<br/> Requires(rpmlib): rpmlib(CompressedFileNames) &lt;= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) &lt;= 4.0-1<br/> Checking for unpackaged file(s): /usr/lib/rpm/check-files /tmp/rpmbuild-lustre--lud8gVUp/BUILDROOT/lustre-2.8.53_27_ga8b7e8c-1.x86_64<br/> error: Installed (but unpackaged) file(s) found:<br/> /lib/modules/3.12.57-60.35_lustre-default/extra/lustre-osd-ldiskfs/fs/ldiskfs.ko<br/> /lib/modules/3.12.57-60.35_lustre-default/extra/lustre-osd-ldiskfs/fs/osd_ldiskfs.ko<br/> /lib/modules/3.12.57-60.35_lustre-default/extra/lustre-tests/fs/llog_test.ko<br/> /lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/fid.ko<br/> /lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/fld.ko<br/> /lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/lfsck.ko<br/> /lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/llite_lloop.ko<br/> /lib/modules/3.12.57-60.35_lustre-default/extra/lustre/fs/lmv.ko</p> <p>It looks like there were problems with node provisioning step in the most recent revision of the patch. Are you working on a solution for that too?</p> <p>Yes, I am. I will make sure this patch get tested</p> <p>the latest issue is on zfs el7, can't load osd-zfs</p> <p>May 29 08:04:34 onyx-42vm3 mrshd<span class="error">&#91;7705&#93;</span>: root@onyx-42vm5.onyx.hpdd.intel.com as root: cmd='(PATH=$PATH:/usr/lib64/lustre/utils:/usr/lib64/lustre/tests:/sbin:/usr/sbin; cd /usr/lib64/lustre/tests; LUSTRE="/usr/lib64/lustre" sh -c "mkdir -p /mnt/mds1; mount -t lustre #011#011 lustre-mdt1/mdt1 /mnt/mds1");echo XXRETCODE:$?'<br/> May 29 08:04:37 onyx-42vm3 kernel: LNet: HW CPU cores: 2, npartitions: 1<br/> May 29 08:04:37 onyx-42vm3 kernel: alg: No test for adler32 (adler32-zlib)<br/> May 29 08:04:37 onyx-42vm3 kernel: alg: No test for crc32 (crc32-table)<br/> May 29 08:04:46 onyx-42vm3 kernel: Lustre: Lustre: Build Version: 2.8.53_28_g7808b96<br/> May 29 08:04:46 onyx-42vm3 kernel: LustreError: 158-c: Can't load module 'osd-zfs'<br/> May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(genops.c:318:class_newdev()) OBD: unknown type: osd-zfs<br/> May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(obd_config.c:370:class_attach()) Cannot create device lustre-MDT0000-osd of type osd-zfs : -19<br/> May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(obd_mount.c:198:lustre_start_simple()) lustre-MDT0000-osd attach error -19<br/> May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(obd_mount_server.c:1809:server_fill_super()) Unable to start osd on lustre-mdt1/mdt1: -19<br/> May 29 08:04:46 onyx-42vm3 kernel: LustreError: 7740:0:(obd_mount.c:1450:lustre_fill_super()) Unable to mount (-19)<br/> May 29 08:04:46 onyx-42vm3 xinetd<span class="error">&#91;915&#93;</span>: EXIT: mshell status=0 pid=7704 duration=12(sec)<br/> May 29 08:04:46 onyx-42vm3 systemd-logind: Removed session c35.</p> <p>I have checked manually with el6.7 but not el7. I'll check on el7</p> <p>the reason is on el7, somehow another 'x86_64' string was added in the path</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">D: %post(kmod-lustre-osd-zfs-2.8.53_28_g7808b96-3.10.0_327.13.1.el7_lustre.x86_64.x86_64): scriptlet start D: %post(kmod-lustre-osd-zfs-2.8.53_28_g7808b96-3.10.0_327.13.1.el7_lustre.x86_64.x86_64): execv(/bin/sh) pid 25714 + <span class="code-quote">'['</span> -e /boot/<span class="code-object">System</span>.map-3.10.0-327.13.1.el7_lustre.x86_64.x86_64.x86_64 <span class="code-quote">']'</span> + modules=($(find /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64.x86_64.x86_64/extra/lustre-osd-zfs | grep <span class="code-quote">'\.ko$'</span>)) ++ grep <span class="code-quote">'\.ko$'</span> ++ find /lib/modules/3.10.0-327.13.1.el7_lustre.x86_64.x86_64.x86_64/extra/lustre-osd-zfs find: ‘/lib/modules/3.10.0-327.13.1.el7_lustre.x86_64.x86_64.x86_64/extra/lustre-osd-zfs’: No such file or directory </pre> </div></div> <p>Can you please provide a pointer to the failed test in maloo?</p> <p>actually I had to run rpm manually to get that error message. I thought your patch from <a href="https://jira.whamcloud.com/browse/LU-7643" title="Remove kernel version string from Lustre release field" class="issue-link" data-issue-key="LU-7643"><del>LU-7643</del></a> would solve it but it still failed</p> <p>I wonder if this is related to kmodtool. I think zfs/spl is using a later version than the one provided with the distro. should lustre do the same?</p> <p>It does not surprise me that <a href="https://jira.whamcloud.com/browse/LU-7643" title="Remove kernel version string from Lustre release field" class="issue-link" data-issue-key="LU-7643"><del>LU-7643</del></a> does not help.</p> <p>With a stock rhel7.2 kernel with zfs, and I cannot reproduce this problem. So I am guessing that this is another Intel build system problem.</p> <p>I think the first thing I would try is adding a %{warn} statement to the line after "%{!?kernel_version: %global kernel_version %kversion}" in the lustre spec file to print out the value of %kernel_version. That warn statement might show the weird value right there, and then you can work backward from that to find where it originates.</p> <p>yeah, so the kernel_version print out is "3.10.0-327.18.2.el7.x86_64"<br/> This sounds like we need to build lustre kmod using kmods 2 (similar to spl/zfs using generic spec)</p> <p>I think it is too early to give up on this approach. Lets keep digging.</p> <p>On RHEL7, /usr/lib/rpm/redhat/kmodtool has the following function:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">get_kernel_release () { <span class="code-keyword">if</span> [[ -z $1 ]]; then uname -r <span class="code-keyword">return</span> fi local arch=$(arch) local verrel=${1%.$arch} local verprefix=${verrel%.*} local versuffix=${verrel#$verprefix} verrel=$(ls -Ud /usr/src/kernels/$verprefix*$versuffix.$arch | sort -V | tail -n 1) verrel=${verrel##*/} [[ -z $verrel ]] &amp;&amp; verrel=$1.$arch echo <span class="code-quote">"$verrel"</span> } </pre> </div></div> <p>This is no doubt where the problem arises for lbuild on rhel7. lbuild, as we all know, employees a very non-standard build environment. Since the kernel is not in the correct location during packaging, this function is falling back to setting verrel to "$1.$arch". That is almost certainly where that extra ".x86_64" is coming from, but only under lbuild on rhel7.</p> <p>The trick here will be to come up with more hackery for lbuild that allows the lustre.spec file to remain relatively clean for proper build environments.</p> <p>Meanwhile, Minh, do you have a plan to fix the Intel buildfarm to install the correct packages when this patch is in use? The node provisioning step still seems to always install the old packages, which winds up pulling in part kmod and part dkms packages.</p> <p>Chris, actually I have fixed the one under Onyx cluster. We are rolling out to the rest of the clusters. I checked and the installation has worked on Onyx. Unfortunately, most of the testing went to Trevis cluster in the last few days (week) due to Onyx being loaded with master testing.</p> <p>OK, great. If you have that figured out, then we could be in good shape.</p> <p>I believe that I have hacked around lbuild's latest problem on RHEL7. I tested my hack in <a href="http://review.whamcloud.com/20572" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/20572</a>, and it seemed to work. No triple ".x86_64" in the scripts, and it passed testing...but that was on trevis. I don't know why it passed there despite failing ldiskfs package installation. Very strange.</p> <p>I squashed my work-around into Patch Set 44 of this ticket's patch, <a href="http://review.whamcloud.com/12063" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12063</a>.</p> <p>Are there any other known problems?</p> <p>actually, I also have a version and I am testing right now. <a href="http://review.whamcloud.com/#/c/18426/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/18426/</a><br/> Please take a look while I continue to test</p> <p>The patch is apparently hung up on unrelated test failures now.</p> <p>It would still be good to get reviews in now, so we can get this landed as soon as reasonable.</p> <p>I found in build log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ echo <span class="code-quote">'The kernel ABI reference files (provided by kabi-whitelists) were not found.'</span> The kernel ABI reference files (provided by kabi-whitelists) were not found. + echo <span class="code-quote">'No compatibility check was performed. Please install the kABI reference files'</span> No compatibility check was performed. Please install the kABI reference files + echo <span class="code-quote">'and rebuild <span class="code-keyword">if</span> you would like to verify compatibility with kernel ABI.'</span> and rebuild <span class="code-keyword">if</span> you would like to verify compatibility with kernel ABI. + echo '' </pre> </div></div> <p>we need to install kabi-whitelists rpm don't we?</p> <p>It helps, but Lustre uses symbols not the in the whitelist, so you will still get a warning.</p> <p>Does it still? <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> I know this was the case when I tried to make all of this work with RHEL many years ago.</p> <p>I don't this we'll get the warning after install the kabi-whitelist rpm. I did it locally and didn't see that. we'll update the builders and see</p> <p>I just tried the latest patch and I don't see lustre-module-* anymore. Is this expected?</p> <p>I also tried installing them RPMs and got:</p> <p>rpm <del>Uvh --force tmp/RPMS/lustre/lustre</del>*<br/> error: Failed dependencies:<br/> kmod-lustre = 2.8.54_60_g2a55f34_dirty is needed by lustre-2.8.54_60_g2a55f34_dirty-2.6.32_573.12.1.el6.head.x86_64.x86_64<br/> kmod-lustre = 2.8.54_60_g2a55f34_dirty is needed by lustre-tests-2.8.54_60_g2a55f34_dirty-2.6.32_573.12.1.el6.head.x86_64.x86_64</p> <p>James,<br/> If you carefully read the commit header it says how the packages have been renamed.</p> <p>Thank you for picking this up and following through with it.</p> <p>My complaint is that I'm still the owner and can't give you a +1!</p> <p>Edit: I won't bother nitpicking changes to lbuild.</p> <p>Okay, for some reason my initial build didn't produce the kmod-* rpms. I had to delete the source tree and do it over again. Now I see</p> <p>ksym(snprintf) = 0x9edbecae is needed by kmod-lustre-2.8.54_60_g2a55f34_dirty-2.6.32_573.12.1.el6.head.x86_64.x86_64<br/> ksym(sscanf) = 0x42224298 is needed by kmod-lustre-2.8.54_60_g2a55f34_dirty-2.6.32_573.12.1.el6.head.x86_64.x86_64<br/> ....</p> <p>When I go to install kmod-lustre-*.x86_64.rpm</p> <p>James, are you able to build and install it. Please let me know if you have any problem. if not could you re-review that patch?</p> <p>Nope, but looking at how I build our kernel I might know why. When building the kernel rpm I used --without kabichk. Would that be the reason it doesn't work? This is all done using RHEL6.7.</p> <p>Remove --without kabichk and still gives me the ksym issues.</p> <p>Is the target kernel installed on the build system (or environment)?<br/> Can you verify that dependency generation is done with the right kernel?</p> <p>No the target kernel rpm is not installed on the build system. That would be a big no no since the machine is used for other purposes. The target kernel source tree is installed in a special directory which is not /usr/src to build against. I will be looking into why the dependency generation is done against the wrong kernel. I wonder if we have to do a OFED style build process in which we tell the location of the symver.</p> <p>So I created the most basic test to show the problem. This is with building just the patchless client. Currently our build machine is at RHEL6.7 and my test node is running RHEL6.8. So we have very different kernels running on both. On the build machine I as non-root installed the RHEL6.8 development tree in /tmp so I have /tmp//usr/src/kernels/2.6.32-642.1.1.el6.x86_64. Then I went into my lustre tree containing this patch and did:</p> <p>cd lustre_release<br/> ./autogen.sh<br/> ./configure --disable-server --with-linux=/tmp/usr/src/kernels/2.6.32-642.1.1.el6.x86_64<br/> make rpm</p> <p>Then I went to install the rpms into the image and got the ksyms errors still. This points to the current patch for <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> breaks --with-linux for configure. Note this is very common process our admins use to prepare our client rpms. You can say well just build on the local node. In that case the patch for <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> need to be updated to remove --with-linux support. Also if you recommend using mock to work around this problem then please update this patch to ensure configure failed unless ./configure runs in a mock environment and the source rpm has a hard dependency on mock. In any case I have a reproducer of this problem.</p> <p>it worked on el7. I'll verify on el6 as you did next</p> <p><span class="error">&#91;root@onyx-21vm5 lustre-release&#93;</span># uname -a<br/> Linux onyx-21vm5.onyx.hpdd.intel.com 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux<br/> sh ./autogen.sh<br/> ./configure --disable-server --with-linux=/usr/src/kernels/3.10.0-327.18.2.el7.x86_64/<br/> make rpms</p> <p>scp kmod-lustre-client-2.8.54_61_gcc7a8c9-3.10.0_327.18.2.el7.x86_64.x86_64.rpm root@onyx-24:/root<br/> scp lustre-client-2.8.54_61_gcc7a8c9-3.10.0_327.18.2.el7.x86_64.x86_64.rpm root@onyx-24:/root</p> <p><span class="error">&#91;root@onyx-24 ~&#93;</span># uname -a<br/> Linux onyx-24.onyx.hpdd.intel.com 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux<br/> <span class="error">&#91;root@onyx-24 ~&#93;</span># rpm -hiv ./kmod-lustre-client-2.8.54_61_gcc7a8c9-3.10.0_327.18.2.el7.x86_64.x86_64.rpm<br/> Preparing... ################################# <span class="error">&#91;100%&#93;</span><br/> Updating / installing...<br/> 1:kmod-lustre-client-2.8.54_61_gcc7################################# <span class="error">&#91;100%&#93;</span><br/> <span class="error">&#91;root@onyx-24 ~&#93;</span># modprobe lustre<br/> <span class="error">&#91;root@onyx-24 ~&#93;</span># modinfo lustre<br/> filename: /lib/modules/3.10.0-327.18.2.el7.x86_64/extra/lustre-client/fs/lustre.ko<br/> license: GPL<br/> version: 2.8.54_61_gcc7a8c9<br/> description: Lustre Client File System<br/> author: OpenSFS, Inc. &lt;<a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a>&gt;<br/> rhelversion: 7.2<br/> srcversion: F6CD34A134CF6E45A10B0DB<br/> depends: obdclass,ptlrpc,libcfs,lnet,lmv,mdc,lov<br/> vermagic: 3.10.0-327.18.2.el7.x86_64 SMP mod_unload modversions <br/> <span class="error">&#91;root@onyx-24 ~&#93;</span># </p> <p>Okay. Lets see if its a RHEL6.X issue. Especially since I don't have access to RHEL7 systems <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> If you manage to build it then the build process must be dependent on something missing on our build machine. If I see this problem other people will to once its released into the wild.</p> <p>I don't see any issue on el6. I built on el6.7 pointing to el6.8 kernel and install kmod lustre on el6.8 system. no issue</p> <p><span class="error">&#91;root@onyx-21vm2 lustre-release&#93;</span># uname -a<br/> Linux onyx-21vm2.onyx.hpdd.intel.com 2.6.32-573.26.1.el6.x86_64 #1 SMP Wed May 4 00:57:44 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux<br/> sh autogen.sh <br/> ./configure --disable-server --with-linux=/usr/src/kernels/2.6.32-642.1.1.el6.x86_64/<br/> make rpms<br/> scp kmod-lustre-client-2.8.54_61_gcc7a8c9-2.6.32_642.1.1.el6.x86_64.x86_64.rpm onyx-21vm3:/root<br/> scp lustre-client-2.8.54_61_gcc7a8c9-2.6.32_642.1.1.el6.x86_64.x86_64.rpm onyx-21vm3:/root<br/> history</p> <hr /> <p> rpm -hiv ./kmod-lustre-client-2.8.54_61_gcc7a8c9-2.6.32_642.1.1.el6.x86_64.x86_64.rpm <br/> <span class="error">&#91;root@onyx-21vm3 ~&#93;</span># modprobe lustre<br/> LNet: HW CPU cores: 1, npartitions: 1<br/> alg: No test for adler32 (adler32-zlib)<br/> alg: No test for crc32 (crc32-table)<br/> Lustre: Lustre: Build Version: 2.8.54_61_gcc7a8c9<br/> LNet: Added LNI 10.2.4.14@tcp <span class="error">&#91;8/256/0/180&#93;</span><br/> LNet: Accept secure, port 988<br/> <span class="error">&#91;root@onyx-21vm3 ~&#93;</span># modinfo lustre<br/> filename: /lib/modules/2.6.32-642.1.1.el6.x86_64/extra/lustre-client/fs/lustre.ko<br/> license: GPL<br/> version: 2.8.54_61_gcc7a8c9<br/> description: Lustre Client File System<br/> author: OpenSFS, Inc. &lt;<a href="http://www.lustre.org/" class="external-link" target="_blank" rel="nofollow noopener">http://www.lustre.org/</a>&gt;<br/> srcversion: F6CD34A134CF6E45A10B0DB<br/> depends: obdclass,ptlrpc,libcfs,lnet,lmv,mdc,lov<br/> vermagic: 2.6.32-642.1.1.el6.x86_64 SMP mod_unload modversions <br/> <span class="error">&#91;root@onyx-21vm3 ~&#93;</span># uname -a<br/> Linux onyx-21vm3.onyx.hpdd.intel.com 2.6.32-642.1.1.el6.x86_64 #1 SMP Tue May 31 21:57:07 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux<br/> <span class="error">&#91;root@onyx-21vm3 ~&#93;</span># </p> <p>I am not sure what's missing. I setup the builder without anything special. installed git, libtool, rpm-build</p> <p>How did you put the kernel-devel on the builder? cpio or rpm install?</p> <p>Are you ssh into the destination node and then installing the rpm on that node? For our systems we create the rpms and chroot into a directory that serves as the root of my image for the diskless test nodes.</p> <p>[ management server ] -&gt; /export-image/root -&gt; diskless-node:/root<br/> /usr /usr</p> <p>chroot /export-image<br/> rpm -ivh /tmp/<b>lustre</b>.rpm</p> <p>Could the chroot environment be causing the ksym issues? In our diskless setup its not really possible to install rpms directly on the test nodes since they are essentially read only. Can you try that setup Minh please.</p> <p>"Are you ssh into the destination node and then installing the rpm on that node?"<br/> yes</p> <p>I am not familiar with chroot. For diskless node, I know that LLNL Chaos built the image with the set of rpms, then refresh/boot the node. I'll have to look into chroot more; but I am pretty sure that's the difference and causing the issue.</p> <p>Can you attach or upload your kmod-lustre-client rpm so I can check its content?</p> <p>I found the source of the problem. This change now requires the package kabi-whitelist at least for RHEL6. Do you have this package for RHEL7 and SLES as well? Once I installed kaki-whitelist it appears to work. Well ZFS still gives trouble but it might be a simple case of rebuilding it.</p> <p>I'm back from work travel, so I can jump back into the conversation now.</p> <p>James, I don't believe that kernel-abi-whitelists is required on either RHEL6 or RHEL7. Yes, there will be a warning about it being missing at build time, but that doesn't hurt the packaging process. Clean RHEL6.7 and RHEL7.2 images have been my main testing platforms (with some SLES 12.X on occasion). I checked both of my RHEL images just now, and neither have kernel-abi-whitelists installed. The rpm packages generated under this patch install just fine, and appear to have all required kernel symbols expressed in the rpm --requires section.</p> <p>By the way, if your packages really lists this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ksym(snprintf) = 0x9edbecae ksym(sscanf) = 0x42224298 </pre> </div></div> <p>then there is an additional problem that I didn't remember off the top of my head on the road. Those should not be "ksym" requirements; they should be "kernel" requirements. I would check the package with "rpm -qp --requires" and see how they look. Here are the requirements from the kmod-lustre package on my RHEL6.7<br/> system:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>kernel(snprintf) = 0x9edbecae kernel(sscanf) = 0x42224298 </pre> </div></div> <p>A "kernel" requirement on RHEL means that the RH build system knows that the symbol is provided by the kernel package itself. Any other kernel symbol provided by some external package (e.g. zfs) will be a "ksym" requirement instead of "kernel". So this still implies that Lustre was not built on a system with the prerequisite kernel rpms properly installed.</p> <p>FYI, SUSE does not have a kernel-abi-whitelists package. They do not even employ the approach of having a symbol whitelist.</p> <p><a href="https://drivers.suse.com/doc/SolidDriver/SUSE_Kernel_ABI_Stability.html#does-suse-provide-a-kernel-abi-symbol-whitelist" class="external-link" target="_blank" rel="nofollow noopener">https://drivers.suse.com/doc/SolidDriver/SUSE_Kernel_ABI_Stability.html#does-suse-provide-a-kernel-abi-symbol-whitelist</a></p> <p>Minh said:</p> <blockquote><p>we need to install kabi-whitelists rpm don't we?</p></blockquote> <p>That is not a requirement for this ticket. It probably won't hurt, but you can track that activity in a separate ticket if you like. It does not need to be linked to this issue.</p> <p>In reply to James from gerrit:</p> <blockquote><p>I don't modify any RHEL build scripts. Its a simple rpm extract source and build. Nothing fancy. Forcing people to require things like mock to just build client rpms is not going to go over well with sites.</p></blockquote> <p>No one is requiring special things like mock. We just require that, when building rpms, the prerequisite BuildRequires must actually be installed. This is a completely normal thing to expect in the rpm packaging world, and it is what all of the behind-the-scenes scripts from the distro assume will be the case. Most people will be able to meet that requirement, I think. To avoid installing the prerequisites, you have to replace system-provided behind-the-scenes scripts with your own. We did that for lbuild, despite how icky I felt doing it. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>. If you refuse to meet that simple, accepted requirement, you are free to hack up your own solution as well.</p> <p>I just recommend mock for the places where you don't have access to a system where the BuildRequires are installed, or you don't have the permissions to install those packages. mock is designed exactly for those situations: it sets up a clean base chroot environment, and then installs only the packages required by the source rpm's BuildRequires.</p> <p>If none of <em>those</em> things will work for people, they can always just download prebuilt binaries. The prebuilt binaries are going to work on a much wider set of kernels once they are no longer tied to one specific kernel version.</p> <p>Yes, some adaptation of process will be necessary for some people. I know, change is hard. I totally feel your pain. The Lustre community has gotten very used to its oddball way of building and packaging. But that means that Lustre just flat out doesn't build through the standard tools of major distros like Fedora and RHEL. I think the advantage of supporting those standard methods vastly outweighs the short term pain we will feel as we adapt our personal processes to the new packaging methods.</p> <p>I discovered the kabi-whitelist issues with the following error during the build process:</p> <p>Finding Provides: /usr/lib/rpm/redhat/find-provides<br/> KERNEL ABI COMPATIBILITY WARNING<br/> The kernel ABI reference files (provided by kabi-whitelists) were not found.<br/> No compatibility check was performed. Please install the kABI reference files<br/> and rebuild if you would like to verify compatibility with kernel ABI.</p> <p>Finding Requires: /usr/lib/rpm/redhat/find-requires</p> <p>Once I installed kabi-whitelish and the rpms started to work. Also 'rpm <del>qp --release kmod-lustre</del>*' prints out</p> <p>kernel(__wake_up) = 0x642e54ac<br/> kernel(add_wait_queue) = 0x650fb346<br/> kernel(copy_from_user) = 0x3302b500<br/> kernel(default_wake_function) = 0xffd5a395<br/> kernel(kfree) = 0x037a0cba</p> <p>So its working correct now.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/12063/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/12063/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-5614" title="use %kernel_module_package for weak-updates" class="issue-link" data-issue-key="LU-5614"><del>LU-5614</del></a> build: use %kernel_module_package in rpm spec<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3b7d27ea22faf1c6d0a37afa724fd9b5c3240322</p> <p>The patch landed to master, so I closing this ticket. Good work everyone!</p> <p>If there is any fallout that we need to work through, we can open new tickets for that.</p> <p>yes, a small fix for suse12 is in <a href="https://jira.whamcloud.com/browse/LU-8343" title="build failures on SLES" class="issue-link" data-issue-key="LU-8343"><del>LU-8343</del></a></p> <p>FYI:<br/> <a href="http://cdn.opensfs.org/wp-content/uploads/2016/04/LUG2016D1_Improved-Versioning_Morrone_DiNatale.pdf" class="external-link" target="_blank" rel="nofollow noopener">http://cdn.opensfs.org/wp-content/uploads/2016/04/LUG2016D1_Improved-Versioning_Morrone_DiNatale.pdf</a><br/> <a href="https://www.youtube.com/watch?v=50VN9Q7BHYA&amp;index=5&amp;list=PLA5dHg1_l3V8r6eqrUnl8DiB-KH1K5yLH" class="external-link" target="_blank" rel="nofollow noopener">https://www.youtube.com/watch?v=50VN9Q7BHYA&amp;index=5&amp;list=PLA5dHg1_l3V8r6eqrUnl8DiB-KH1K5yLH</a></p> Blocker LU-7643 Related LU-7722 LU-8377 LU-3956 LU-3953 LU-8343 LU-8519 LU-8383 LU-6677 Development End date Tue, 7 Jun 2016 04:15:18 +0000 Rank 1|hzww3r: Rank (Obsolete) 15704 Severity Start date Fri, 12 Sep 2014 04:15:18 +0000