https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-3121 Lustre <p>This issue was created by maloo for Li Wei &lt;liwei@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://maloo.whamcloud.com/test_sets/418e2830-9f9f-11e2-9f27-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/418e2830-9f9f-11e2-9f27-52540035b04c</a>.</p> <p>The sub-test lustre-initialization_1 failed with the following error:</p> <blockquote> <p>Test system failed to start single suite, so abandoning all hope and giving up</p></blockquote> <p>Info required for matching: lustre-initialization-1 lustre-initialization_1</p> <p>For some unknown reason, the MDS couldn't load osd-zfs.ko:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>08:17:45:LNet: Accept all, port 7988 08:17:45:LustreError: 158-c: Can't load module 'osd-zfs' 08:17:45:LustreError: 2944:0:(genops.c:304:class_newdev()) OBD: unknown type: osd-zfs 08:17:45:LustreError: 2944:0:(obd_config.c:374:class_attach()) Cannot create device lustre-MDT0000-osd of type osd-zfs : -19 08:17:45:LustreError: 2944:0:(obd_mount.c:196:lustre_start_simple()) lustre-MDT0000-osd attach error -19 08:17:45:LustreError: 2944:0:(obd_mount_server.c:1664:server_fill_super()) Unable to start osd on lustre-mdt1/mdt1: -19 08:17:45:LustreError: 2944:0:(obd_mount.c:1264:lustre_fill_super()) Unable to mount (-19) </pre> </div></div> LU-3121

lustre-initialization-1: Can't load module 'osd-zfs'

Bug Blocker Resolved Fixed Chris Gearing Maloo HB zfs Mon, 8 Apr 2013 02:38:02 +0000 Tue, 2 Jun 2015 20:49:45 +0000 Mon, 5 Aug 2013 15:18:55 +0000 Lustre 2.4.1 0 8 <p>Lai</p> <p>Could you please comment?</p> <p>Thanks</p> <p>Peter</p> <p>I spent some time to provision the build manually on Toro. After "yum install"ing lustre-osd-zfs RPM and the ZFS RPMs, the MDT could be successfully mounted. This suggests something might be wrong with Autotest/Toro.</p> <p>Duplicate of <a href="https://jira.whamcloud.com/browse/LU-2969" title="Fail to mount MDS as zfs on external OFED build" class="issue-link" data-issue-key="LU-2969"><del>LU-2969</del></a></p> <p>Seems to only hit rarely but we'd still like to understand why this happens</p> <p>I will had extra debug to Autotest which will require a restart, this issue must be intermittent because zfs testing does run.</p> <p><a href="https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&amp;test_group=review-zfs&amp;commit=Apply+Filter" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions?utf8=%E2%9C%93&amp;test_group=review-zfs&amp;commit=Apply+Filter</a></p> <p>I think the <a href="https://maloo.whamcloud.com/test_sessions/3ef2158c-9f9f-11e2-9f27-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/3ef2158c-9f9f-11e2-9f27-52540035b04c</a> failure is due to the patch being tested.</p> <p>Nathaniel, could you explain how the patch caused the failure, please? I went through the patch again, but did not figure out why it would cause such failures.</p> <p>Li, I am wildly incorrect. I must have been looking at a different patch. My comment form the 9th is wrong. Sorry for the confusion.</p> <p>No problem. Ironically, my <a href="http://review.whamcloud.com/5785" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5785</a> has been hitting this issue every time recently, although I can't see any fault in the patch itself. Probably there really is a problem in the patch... <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Ah, there are also considerable amount of failures when testing other patches.</p> <p>another hit at <a href="https://maloo.whamcloud.com/test_sets/d170d43c-a767-11e2-b3cc-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/d170d43c-a767-11e2-b3cc-52540035b04c</a></p> <p>Another instance: <a href="https://maloo.whamcloud.com/test_sets/2352a39e-b658-11e2-bf90-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/2352a39e-b658-11e2-bf90-52540035b04c</a></p> <p>This is blocking the patch review testing on zfs:<br/> <a href="https://maloo.whamcloud.com/test_sets/f3082170-b678-11e2-a5af-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/f3082170-b678-11e2-a5af-52540035b04c</a></p> <p>Another one: <a href="https://maloo.whamcloud.com/test_sets/cd712682-b72c-11e2-bd0f-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/cd712682-b72c-11e2-bd0f-52540035b04c</a></p> <p>rpms seem to install and run cleanly, must be an install issue of some sort. Possibly depmod issue?</p> <p>I can reproduce these exact symptoms by installing all of lustre except lustre-osd-zfs normally, then install lustre-osd-zfs with --noscripts (causing it to skip running depmod). lustre-osd-zfs may skip depmod if the kernel isn't installed yet, still trying to figure out this exact vector.</p> <p>If it's a module dependency ordering issue, it may be solved by <a href="http://review.whamcloud.com/6259" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6259</a> which fixes up lustre-osd-* / lustre-modules rpm dependencies.</p> <p>The above linked patch has landed for <a href="https://jira.whamcloud.com/browse/LU-3269" title="Many symbol warnings at rpm installation time" class="issue-link" data-issue-key="LU-3269"><del>LU-3269</del></a>. </p> <p>Lustre Branch: master<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-master/1495" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1495</a> (which contains the patch for <a href="https://jira.whamcloud.com/browse/LU-3269" title="Many symbol warnings at rpm installation time" class="issue-link" data-issue-key="LU-3269"><del>LU-3269</del></a>)<br/> FSTYPE=zfs</p> <p>The issue in this ticket still occurred:<br/> <a href="https://maloo.whamcloud.com/test_sessions/754709ca-bd37-11e2-a548-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sessions/754709ca-bd37-11e2-a548-52540035b04c</a></p> <p>Lustre Tag: v2_4_0_RC1<br/> Lustre Build: <a href="http://build.whamcloud.com/job/lustre-master/1501/" class="external-link" target="_blank" rel="nofollow noopener">http://build.whamcloud.com/job/lustre-master/1501/</a><br/> Distro/Arch: RHEL6.4/x86_64<br/> FSTYPE=zfs</p> <p>Hit the issue again:<br/> <a href="https://maloo.whamcloud.com/test_sets/7e89ecee-bf5a-11e2-88e0-52540035b04c" class="external-link" target="_blank" rel="nofollow noopener">https://maloo.whamcloud.com/test_sets/7e89ecee-bf5a-11e2-88e0-52540035b04c</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>+ yum install -y kernel-2.6.32-358.6.1.el6_lustre.x86_64 lustre-ldiskfs lustre-modules lustre lustre-tests Loaded plugins: fastestmirror, security Setting up Install Process Resolving Dependencies --&gt; Running transaction check ---&gt; Package kernel.x86_64 0:2.6.32-358.6.1.el6_lustre will be installed ---&gt; Package lustre.x86_64 0:2.4.50-2.6.32_358.6.1.el6_lustre.x86_64 will be installed --&gt; Processing Dependency: lustre-osd for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64 --&gt; Processing Dependency: libnetsnmpmibs.so.20()(64bit) for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64 --&gt; Processing Dependency: libnetsnmphelpers.so.20()(64bit) for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64 --&gt; Processing Dependency: libnetsnmpagent.so.20()(64bit) for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64 --&gt; Processing Dependency: libnetsnmp.so.20()(64bit) for package: lustre-2.4.50-2.6.32_358.6.1.el6_lustre.x86_64.x86_64 ---&gt; Package lustre-ldiskfs.x86_64 0:4.1.0-2.6.32_358.6.1.el6_lustre.x86_64 will be installed ---&gt; Package lustre-modules.x86_64 0:2.4.50-2.6.32_358.6.1.el6_lustre.x86_64 will be installed ---&gt; Package lustre-tests.x86_64 0:2.4.50-2.6.32_358.6.1.el6_lustre.x86_64 will be installed --&gt; Running transaction check ---&gt; Package lustre-osd-ldiskfs.x86_64 0:2.4.50-2.6.32_358.6.1.el6_lustre.x86_64 will be installed ---&gt; Package net-snmp-libs.x86_64 1:5.5-44.el6_4.1 will be installed --&gt; Processing Dependency: libsensors.so.4()(64bit) for package: 1:net-snmp-libs-5.5-44.el6_4.1.x86_64 --&gt; Running transaction check ---&gt; Package lm_sensors-libs.x86_64 0:3.1.1-17.el6 will be installed --&gt; Finished Dependency Resolution </pre> </div></div> <p>The problem is that lustre requires lustre-osd and picks lustre-osd-ldiskfs thus not installing lustre-osd-zfs</p> <p>The yum line should explicitly install lustre-osd-ldiskfs and lustre-osd-zfs, or we could add a virtual package lustre-all that requires both osd packages.</p> <p>zfs and ancillary packages are installed after kickstart, but lustre-osd-zfs isn't one of them.</p> <p>I believe this is fixed now. There is still some bug where lustre-inialization-1 fails, but it does not appear to be failing on loading osd-zfs.</p> Related LU-2969 Development Rank 1|hzvn9b: Rank (Obsolete) 7578 Severity