https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9745 Lustre <p>When I run the following command to install DKMS built Lustre:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># yum install kernel-devel-[0-9]*_lustre lustre lustre-dkms kmod-lustre-osd-ldiskfs zfs </pre> </div></div> <p>the result after the installation is that <b>only</b> the <tt>lustre.ko</tt> (built from <em>dkms-lustre</em>) module is in <tt>/lib/modules/3.10.0-514.21.1.el7_lustre.x86_64/extra/</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls -l /lib/modules/3.10.0-514.21.1.el7_lustre.x86_64/extra/ total 4836 -rw-r--r-- 1 root root 1615824 Jul 7 14:01 lustre.ko drwxr-xr-x 3 root root 16 Jul 7 13:55 lustre-osd-ldiskfs -rw-r--r-- 1 root root 353632 Jul 7 13:55 splat.ko -rw-r--r-- 1 root root 170024 Jul 7 13:55 spl.ko -rw-r--r-- 1 root root 14016 Jul 7 13:57 zavl.ko -rw-r--r-- 1 root root 75848 Jul 7 13:57 zcommon.ko -rw-r--r-- 1 root root 2205152 Jul 7 13:57 zfs.ko -rw-r--r-- 1 root root 132488 Jul 7 13:57 znvpair.ko -rw-r--r-- 1 root root 34000 Jul 7 13:57 zpios.ko -rw-r--r-- 1 root root 330920 Jul 7 13:57 zunicode.ko </pre> </div></div> <p>Notice that all of the other supporting modules are missing.</p> <p>After the above, if I then remove the module with <tt>dkms uninstall -m lustre/2.10.0_RC1 -k 3.10.0-514.21.1.el7_lustre.x86_64</tt> and then run <tt>/etc/kernel/postinst.d/dkms 3.10.0-514.21.1.el7_lustre.x86_64</tt> to emulate what happens during the yum installation above, <tt>/lib/modules/3.10.0-514.21.1.el7_lustre.x86_64/extra/</tt> contains all of the necessary Lustre modules.</p> <p>So there seems to be some subtle issue with the <em>lustre-dkms</em> RPM that only occurs during the initial installation.</p> LU-9745

dkms-lustre does not install all modules on initial autoinstall

Bug Critical Resolved Fixed Nathaniel Clark Brian Murrell Fri, 7 Jul 2017 14:28:42 +0000 Fri, 18 Aug 2017 23:29:15 +0000 Fri, 18 Aug 2017 21:57:15 +0000 Lustre 2.10.0 Lustre 2.10.1 Lustre 2.11.0 0 7 <p>Nathaniel</p> <p>Can you please work on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Brian,</p> <p>How does your procedure compare to that in the following doc:</p> <p><a href="https://github.com/intel-hpdd/intel-manager-for-lustre/issues/125" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/intel-hpdd/intel-manager-for-lustre/issues/125</a></p> <p>My equivalent command is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">yum --nogpgcheck install \ lustre-dkms \ spl-dkms \ zfs-dkms \ kmod-lustre-osd-ldiskfs \ lustre-osd-ldiskfs-mount \ lustre-osd-zfs-mount \ lustre \ lustre-resource-agents </pre> </div></div> <p>Note that I ended up using DKMS for all the kmods except LDISKFS.</p> <p>Which version of dkms is this with?</p> <blockquote> <p>Which version of dkms is this with?</p></blockquote> <p>Whichever is in EPEL, which seems to be dkms-2.3-5.20170523git8c3065c.el7 currently.</p> <p>I've tried reproducing, but it installed fine for me (installing all zfs/spl then installing lustre). I'm trying to reproduce by installing everything in once yum call.</p> <p>Do you have logs?</p> <p><b>EDIT:</b> I could not reproduce installing everything at once. Can you reproduce? If there were old lustre-dkms version installed, that may be the cause (dkms is really crappy at cleaning up after itself)</p> <p>Here is a <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/27717/27717_transcsript" title="transcsript attached to LU-9745">transcsript^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> demonstrating the problem.</p> <p>The system this was run on is still available if you want any more information from it.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=utopiabound" class="user-hover" rel="utopiabound">utopiabound</a>: Was my transcript helpful?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=brian" class="user-hover" rel="brian">brian</a>,</p> <p>Yes. Usually dkms is installed prior to packages that need dkms. Do you have any logs in <tt>/var/lib/dkms/lustre/2.10.0/*/x86_64/log/</tt> ?</p> <p>I tried reproducing this and I always get a full set of kernel modules.</p> <p>Why not do a Webex or similar to Brian's environment?</p> <blockquote> <p>Yes. Usually dkms is installed prior to packages that need dkms.</p></blockquote> <p>I don't think you can (or should) make that a requirement with RPM. It should not be a requirement. People familiar with RPM are going to do exactly as I have done.</p> <blockquote> <p>Do you have any logs in /var/lib/dkms/lustre/2.10.0/*/x86_64/log/</p></blockquote> <p>I'm afraid I don't. I needed to recycle the machine for other tests.</p> <blockquote> <p>I tried reproducing this and I always get a full set of kernel modules.</p></blockquote> <p>Did you try reproducing <b>exactly</b> as I had done? If so, could you send me a transcript so that I can try to compare it to what I am doing to see if I can find any differences?</p> <p>After trying again.</p> <p>If there is no <tt>kernel-devel</tt> installed prior to the yum command, not all lustre modules are installed.</p> <p>If even an older <tt>kernel-devel</tt> is installed all lustre modules are installed.</p> <p>Looking at why.</p> <p><b>WORKAROUND</b><br/> Install lustre-devel and dkms then, install everything else.</p> <p>Actually, <tt>kernel-devel</tt> is the essential pre-req – DKMS will get pulled in properly without needing to be pre-installed. I'm working on the upgrade / migration process for EE to Lustre 2.10 and have a reliable procedure, where I have documented the installation of the kernel packages as a standalone step. This was done to separate the steps into units that can be easily audited, and to incorporate the required reboot early in the process. I also thought it would be a way to avoid unnecessary rebuilds of the modules for other kernel versions, which seems to have been borne out. However, I'll review this today as I go through the upgrades of the other servers in my testbed.</p> <p>Brian</p> <p>Have you had a chance to test the proposed workaround?</p> <p>Peter</p> <blockquote> <p>Have you had a chance to test the proposed workaround?</p></blockquote> <p>I have not. I was not aware that that was going to be the solution for IML.</p> <p>Having to stage the installation into multiple steps goes in entirely the opposite direction that we are trying to go with IML. It is designed around having a single install target per profile and having to split that up into multiple steps will require a non-insignificant redesign of the host deployment process.</p> <p> </p> <p>I have tried several combinations of installing Lustre 2.10.0 and ZFS 0.6.5.11 with DKMS on RHEL 7.3. Packages are taken from official public repositories. The only successful method I have been able to incorporate is to introduce a reboot between adding the kernel packages and installing the ZFS and Lustre packages with DKMS. If all the packages are included on the same command line or even if they are separated such that the kernel is installed first, the process fails.</p> <p>My starting point is a basic RHEL 7.3 install with the 3.10.0-514.el7 kernel.</p> <p>For example:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">yum --nogpgcheck --disablerepo=base,extras,updates \ --enablerepo=lustre-server-2.10.0 install \ kernel kernel-devel kernel-headers kernel-tools \ kernel-tools-libs kernel-tools-libs-devel yum --nogpgcheck --enablerepo=lustre-server-2.10.0 install \ lustre-dkms spl-dkms zfs-dkms \ kmod-lustre-osd-ldiskfs \ lustre-osd-ldiskfs-mount lustre-osd-zfs-mount \ lustre lustre-resource-agents ... [root@ct68-oss4 ~]# dkms status lustre, 2.10.0: added spl, 0.6.5.11: added zfs, 0.6.5.11: added </pre> </div></div> <p>At this stage, the DKMS packages are added, but DKMS has not installed the modules. After reboot, DKMS builds and installs the modules as a background task:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@ct68-oss4 ~]# dkms status lustre, 2.10.0: added spl, 0.6.5.11, 3.10.0-514.21.1.el7_lustre.x86_64, x86_64: installed zfs, 0.6.5.11: added </pre> </div></div> <p>On completion, the state is as follows</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[root@ct68-oss4 ~]# dkms status lustre, 2.10.0, 3.10.0-514.21.1.el7_lustre.x86_64, x86_64: installed (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) spl, 0.6.5.11, 3.10.0-514.21.1.el7_lustre.x86_64, x86_64: installed zfs, 0.6.5.11, 3.10.0-514.21.1.el7_lustre.x86_64, x86_64: installed </pre> </div></div> <p>The Lustre modules are build in <tt>/var/lib/dkms</tt>, but do not get copied into <tt>/lib/modules/.../extra/</tt>.</p> <p>I've tried a few variations on this instruction. The only successful outcome was to install the kernel packages, reboot on the new kernel, then continue installation. This process works with a high degree of confidence.</p> <p> </p> <p>Also don't forget the work-around in my original issue description:</p> <blockquote> <p>remove the module with dkms uninstall -m lustre/2.10.0_RC1 -k 3.10.0-514.21.1.el7_lustre.x86_64 and then run /etc/kernel/postinst.d/dkms 3.10.0-514.21.1.el7_lustre.x86_64 to emulate what happens during the yum installation above, /lib/modules/3.10.0-514.21.1.el7_lustre.x86_64/extra/ contains all of the necessary Lustre modules.</p></blockquote> <p>Maybe this helps in debugging the root issue.</p> <p>The problem with this work-around, of course, is that it results in a second, unnecessary build of the Lustre modules, even though they actually already exist in <tt>/var/lib/dkms/...</tt>. I have found no way to ask DKMS to simply install the already built modules instead of first rebuilding them again, before trying to install them.</p> <p>This second, unnecessary rebuild has an end-user impact of the deploying of IML storage servers taking even longer than the already perceived long time it takes.  We have had users complain about the amount of time it takes to deploy storage nodes as it is.</p> <p>I am a little confused about <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=malkolm" class="user-hover" rel="malkolm">malkolm</a>'s work-around though.  I believe DKMS' <em>auto-install</em> option was enabled for all of the modules, but <em>lustre</em> in particular.  That should mean that after installation, even before booting to the <em>lustre</em> kernel, the DKMS modules will be built for it since it was installed with the DKMS modules.  In fact I cannot see what is different between <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=malkolm" class="user-hover" rel="malkolm">malkolm</a>'s work-around and my original reproducer except that I did not examine the kernel module installation state after the reboot.</p> <p>Maybe the reboot always fixes this problem, again at a cost of building the modules twice.</p> <p>It all works the first time, if you have the booted kernels <tt>kernel-devel</tt> package installed.</p> <p>Okay, this worked, and it satisfies the "only build once" requirement, and it does a single reboot (which is required anyway):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>yum install kernel-devel-[0-9]*_lustre kernel-[0-9]*_lustre reboot </pre> </div></div> <p>wait...</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>yum -y install lustre lustre-dkms kmod-lustre-osd-ldiskfs zfs </pre> </div></div> <p>Right.  But it doesn't fit IML's current process of "install (a) package(s), then reboot", which is the current "hardcoded" (for all intents and purposes) process.</p> <p>It's not trivial for us to change that process from a (data driven – meaning the data structure describing that process as well as the code to implement the data structures' instruction) single (installation) step process to multiple (installation) steps separated by a reboot.  Certainly not at this stage of trying to get to a 4.0 release of IML out there.</p> <p>This work-around also becomes "knowledge" that everyone else trying to deploy DKMS-built Lustre systems needs to know about.  Perhaps in could be in Release Notes or something, but does anyone really read those?  Particularly people already familiar with Lustre?</p> <p>So back to the root cause... Have we raised this issue with the DKMS maintainers?  Perhaps this is a trivial issue that somebody intimately familiar with the DKMS code base will immediately recognise.</p> <p>It's weird that zfs/spl both install all of their modules, but lustre doesn't.  It feels like a dkms issue, but there might be a way to mitigate on our end.</p> <p>The process Nathaniel outlined is the one I also described, and it is the only mechanism that I was successfully able to incorporate (in my command line, the first yum command installs lustre-patched kernel packages from a lustre 2.10 repo).</p> <p>Adding the lustre kernel packages and then continuing <em>without</em> a reboot, regardless of combination, leaves the system in the incomplete state when the host is eventually rebooted, where the modules get compiled, but not installed. I'd also point out that the modules are "added" but not installed when the attempt is made without rebooting first.</p> <p>To build on Nathaniel's comment, one possible compromise could be to install the kernel-devel package of the "starting" kernel, as well as the desired kernel, to see if that alters the behaviour prior to reboot. This will cause the modules to be built more than once, but it might be a way to meet the requirement of installing all packages prior to reboot.</p> <p>But, as <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=utopiabound" class="user-hover" rel="utopiabound">utopiabound</a> points out, it's only the Lustre DKMS module that has this problem.  The ZFS/SPL DKMS modules don't have the same problem.</p> <p>So this would appear to be some subtle issue with our module, or at least the way we have written our module is triggering a bug that the ZFL/SPL modules are not.</p> <p>The next step is probably to examine our module and the ZFS/SPL modules closely to see what kind of differences exist between them.</p> <p>I think I know what is going wrong, lustre-dkms uses a "temporary" dkms.conf (which only lists lustre.ko) then rebuilds it during dkms process; I'm not sure why, but I aim to fix it.</p> <p>Nathaniel Clark (nathaniel.l.clark@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28210" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28210</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9745" title="dkms-lustre does not install all modules on initial autoinstall" class="issue-link" data-issue-key="LU-9745"><del>LU-9745</del></a> dkms: Fix included dkms.conf file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 28c700e07d78b04cf7be819c646be2a5b6087301</p> <p>Using <a href="https://review.whamcloud.com/#/c/28210/3/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28210/3/</a> works. It was the bad dkms.conf.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=utopiabound" class="user-hover" rel="utopiabound">utopiabound</a>: Do you mind if I cherry-pick <a href="https://review.whamcloud.com/#/c/28210/" class="external-link" target="_blank" rel="nofollow noopener">this</a> to b2_10 so that I can test it?</p> <p>Brian J. Murrell (brian.murrell@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28224" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28224</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9745" title="dkms-lustre does not install all modules on initial autoinstall" class="issue-link" data-issue-key="LU-9745"><del>LU-9745</del></a> dkms: Fix included dkms.conf file<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 7a2bdccdbb48873a02be70d843c5203ba8c668ca</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/28224/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28224/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9745" title="dkms-lustre does not install all modules on initial autoinstall" class="issue-link" data-issue-key="LU-9745"><del>LU-9745</del></a> dkms: Fix included dkms.conf file<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: a0d855bcf7e55a6c2e0659b5fce7a19a9818d2b0</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28210/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28210/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9745" title="dkms-lustre does not install all modules on initial autoinstall" class="issue-link" data-issue-key="LU-9745"><del>LU-9745</del></a> dkms: Fix included dkms.conf file<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: c99e8035ea543860f6db5e9919ff0045b56d1835</p> <p>Landed for 2.11</p> Development Rank 1|hzzg9b: Rank (Obsolete) 9223372036854775807 Severity