https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14822 Lustre <p>On our MDS running ZFS backing, we're seeing a frequent issue which'll hang the clients and show the attached stack trace.  I'll also attach the lustre-logs for the relevant time period of this morning's instance.</p> <p>Once this has happened, it seems the only option to reconnect the client is a reboot of the MDS.</p> <p>This may be related to the MDT filling up - we changed the zpool topology to increase the size of the MDT and all seemed well for a few days after before these issues started to occur.</p> <p>I'm running an lfsck which has so far repaired a large number of namespaces but the problem as occurred again while that was running.</p> <p>Any help as always much appreciated. </p> EL7 LU-14822

Panic at dnode.c leading to LNet service thread inactive

Bug Minor Resolved Not a Bug Peter Jones Dneg Tue, 6 Jul 2021 16:32:21 +0000 Mon, 12 Jul 2021 16:02:06 +0000 Mon, 12 Jul 2021 16:02:06 +0000 Lustre 2.12.6 0 3 <p>and so you're not just talking to a generic company name, Stephen here <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Stephen</p> <p>What ZFS version are you running? Could this be <a href="https://jira.whamcloud.com/browse/LU-13536" title="Lustre ZFS dnode kernel panic" class="issue-link" data-issue-key="LU-13536"><del>LU-13536</del></a>? If so, moving to a newer ZFS version could address this issue for you.</p> <p>Peter</p> <p>Browsing around it seems this might be related to the issue seen in <a href="https://jira.whamcloud.com/browse/LU-13536" title="Lustre ZFS dnode kernel panic" class="issue-link" data-issue-key="LU-13536"><del>LU-13536</del></a> ?</p> <p>Is there a recommended package set to upgrade to ZFS 0.8.3 or newer/different?  I may of course be barking up the wrong tree here.</p> <p>Hi Peter.  Your comment showed up as I clicked 'Add' <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>  As mentioned, happy to upgrade ZFS.  Do you have a recommended method for that or just grab them from the OpenZFS project?</p> <p>For reference:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>emds1 /tmp # rpm -qa | grep zfs lustre-osd-zfs-mount-2.12.6-1.el7.x86_64 zfs-0.7.13-1.el7.x86_64 kmod-zfs-3.10.0-1160.2.1.el7_lustre.x86_64-0.7.13-1.el7.x86_64 libzfs2-0.7.13-1.el7.x86_64 kmod-lustre-osd-zfs-2.12.6-1.el7.x86_64 </pre> </div></div> <p>Yes - just grab the updated ZFS version from the ZoL site and rebuild. Several sites have done this successfully.</p> <p>Would I be correct in assuming that there's nothing particularly version specific about the two lustre packages in that list (lustre-osd-zfs-mount and kmod-lustre-osd-zfs)?</p> <p>I'll just replace zfs, libzfs2 with the newer ones and swap out the Lustre provided kmod-zfs with zfs-dkms?</p> <p>Had a first pass at this and I'm afraid I think I'm gonna have to ask for some step-by-step here.</p> <p>A quick rpmbuild -ba of the ZFS spec with a simple swap out of the spec file version number and ZFS tar.gz source is taking me down a road of missing files and so on.  Do you have appropriate spec files for the newer versions?  I notice also that the SPL source is no longer separate.  Haven't tried rebuilding that yet.</p> <p>Per the comments in <a href="https://jira.whamcloud.com/browse/LU-13536" title="Lustre ZFS dnode kernel panic" class="issue-link" data-issue-key="LU-13536"><del>LU-13536</del></a>, there are two approaches to solving the crashes in that ticket - updating to ZFS 0.8.x, or patching ZFS 0.7.13 with the two referenced patches:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 78e213946 Fix dnode_hold() freeing dnode behavior 58769a4eb Don’t allow dnode allocation if dn_holds != 0 </pre> </div></div> <p>Due to changes in ZFS between 0.7 and 0.8, if you do a ZFS upgrade you would need to rebuild all of the Lustre RPMs to get a new <tt>kmod-lustre-osd-zfs</tt> since it links directly with the ZFS module. If you apply the patches to the ZFS 0.7.13 you could very likely keep the existing Lustre RPMs since that change is internal only.</p> <p>Right.  Thanks.  Patching for now then.</p> <p>Rebuilt zfs and zfs-dkms with those two patches.  I'll schedule a reboot soon to pick up the patched versions.  Happy for you to close this and I can re-open if necessary or leave it as is for a few days while it's soak tested - whichever is your preference.</p> <p>Great. Why not just let us know after the weekend if there is a noticeable improvement (or sooner of course if there are still problems)?</p> <p>Looking good.  Thanks for your help.</p> <p>Excellent - thanks for the update</p> Development Rank 1|i01yl3: Rank (Obsolete) 9223372036854775807 Severity