https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5525 Lustre <p>After upgrading to lustre 2.4.2-14chaos (see github.com/chaos/lustre), we soon hit the following assertion on one of our MDS nodes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mdt_handler.c:3652:mdt_intent_lock_replace()) ASSERTION( new_lock-&gt;l_readers + new_lock-&gt;l_writers == 0 ) failed</pre> </div></div> <p>Perhaps most significantly, this tag of our lustre tree includes the patch entitled:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LU-4584 mdt: ensure orig lock is found in hash upon resend</pre> </div></div> <p>James Simmons reported this assertion when he tested the <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> patch, but the Bruno <a href="https://jira.hpdd.intel.com/browse/LU-4584?focusedCommentId=82590&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-82590" class="external-link" target="_blank" rel="nofollow noopener">made the evaluation</a> that the assertion was unrelated to the patch.</p> <p>Whether it is related or not, we need to fix the problem.</p> MDS node, Lustre 2.4.2-14chaos, ZFS OBD LU-5525

ASSERTION( new_lock->l_readers + new_lock->l_writers == 0 ) failed

Bug Critical Resolved Duplicate Bruno Faccini Christopher Morrone llnl Wed, 20 Aug 2014 21:55:36 +0000 Wed, 13 Oct 2021 03:03:31 +0000 Wed, 13 Oct 2021 03:03:31 +0000 0 7 <p>Here is the backtrace for the thread that mdt01_006 thread that hit the assertion:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mdt_intent_lock_replace mdt_intent_reint mdt_intent_polucy ldlm_lock_enqueue ldlm_handle_enqueue0 mdt_enqueue mdt_handle_common mds_regular_handle ptlrpc_server_handle_request ptlrpc_main </pre> </div></div> <p>Bruno</p> <p>Can you please advise on this ticket?</p> <p>Thanks</p> <p>Peter</p> <p>Yes, I am investigating the long history around this issue ...</p> <p>We have hit this assertion 3 times in just over 24 hours on the MDS of our largest production filesystem. I reverted the patch for <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> in the hope of getting us back into a reasonably operational state again. We should probably know in a day or two if the problem is gone.</p> <p>Hello Chris,</p> <p>I checked my notes for <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> and this LBUG occured without my patch being applied.</p> <p>I suspect this will not be possible for you, but just in case. Did you ran with some debug levels enabled in Lustre trace at the time of these 3 crashes ? And if yes, could you extract+provide at least one of them ? Or better, could you provide a crash-dump ?</p> <p>Also, and I know it is not an easy question but ..., do you have any idea about what was the work-load that may cause this ?</p> <p>I am trying to setup a platform to reproduce in-house so any details about your configuration is of interest (number of MDSs/MDTs, DNE?, number of OSSs/OSTs, OSTs indexing, default striping, ...).</p> <p>Thanks again and in advance for your help.</p> <p>This is a not a machine for which we can provide logs or crash dumps. No, we don't know the details of the various workloads going on at the time.</p> <p>The filesystem has 768 OSTs on 768 OSS nodes. Default stripe value of 1. Some users do use stripe counts at various widths, some over 700.</p> <p>I don't know what your notes say, but I checked and we crashed the MDS fifteen times during the three days that we were running with the <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> patch. We then rebooted the MDS onto Lustre version 2.4.2-14.1chaos (which does nothing but revert the <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> patch), and we have not seen this ticket's assertion in over three days.</p> <p>While not conclusive, the circumstantial evidence points strongly at the <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> patch either introducing the bug or making it much easier to hit.</p> <p>Here is the list of 2.4.2-14chaos to 2.4.2-14.1chaos changes:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ git log --oneline 2.4.2-14chaos..2.4.2-14.1chaos f28b8cc Revert "LU-4584 mdt: ensure orig lock is found in hash upon resend" </pre> </div></div> <p>You can find those tags at github.com/chaos/lustre.</p> <p>I hit this bug using the latest b2_5 branch with the <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> patch. Good news for you is that I can share my crash dumps. A simple simul run on our Cray test bed produced this Oops. I uploaded the vmcore and dmesg to ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-5525" title="ASSERTION( new_lock-&gt;l_readers + new_lock-&gt;l_writers == 0 ) failed" class="issue-link" data-issue-key="LU-5525"><del>LU-5525</del></a>.</p> <p>Hello James, thanks for this crash-dump already! But I also need the corresponding vmlinux and Lustre modules to allow crash tool to run on it. By "the latest b2_5 branch with the <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> patch", do you mean you used one of our recent Jenkins build to get this crash? If yes, can you point me to which one ? And if not, can you also xfer the 2x kernel-debuginfo<span class="error">&#91;-common&#93;</span> RPMs and also the luste-modules (or the lustre-debuginfo) RPM ?<br/> BTW, I checked the vmcore-dmesg.txt file, you provided with the crash-dump, and its content seems to indicate that you run with both a Kernel and Lustre distro you have builded locally, right?</p> <p>Sorry I forgot the <b>debug</b>.rpms. I just uploaded them to the same ftp spot. ORNL would really like to go to 2.5 ASAP.</p> <p>Yes, I built b2_5 at commit a43e0e4ce4b57240540e8d35a8ba44e203c70ae1 with some additional patches. We tag our kernels to avoid our automated rpm update system from stomping on them. They basically are the RHEL Lustre patched kernels. We also build our own kernel rpms to enable certain things like timestamps in the printk.</p> <p>James, "crash" tool claims "WARNING: kernel version inconsistency between vmlinux and dumpfile" and gives up due to further errors ...<br/> Are you sure that at the time of crash you were running with this "2.6.32-431.17.1.el6.wc" kernel version for which you provided the debuginfo RPMs ? BTW, strings out from vmcore and vmcore-dmesg.txt report "2.6.32-358.23.2.el6.atlas"...</p> <p>Chris, I am really sorry about the fact my b2_4 patch for <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> seems to cause you so many troubles.<br/> This LBUG already has not occured when running auto-tests and your/<a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> reproducer, against Jenkins build for patch-set #7 of Gerrit change #9488, and I am still unable to reproduce it when running with a local build made of tag 2.4.3-RC1 + <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> patch (ie, patch-set #7 of Gerrit change #9488). <br/> I am currently building a Lustre version based on 2.4.2-14chaos and see if I can reproduce in-house, but I wanted to ask you if you tried to run your <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> reproducer against it already ?</p> <p>Oops, you are right. I saw this problem with our 2.4 image which did contain the <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> patch. Sorry for the confusion. I pushed the proper debug rpms this time.</p> <p>Chris, can you help and tell me the kernel version you use/build with 2.4.2-14chaos, I am having problems when patching the 2.6.32-358.23.2 kernel version we use to build our latest b2_4/2.4.3-RC1 branch/tag.</p> <p>I can report I don't see this issue with 2.5 servers. I see another problem instead.</p> <blockquote><p>I wanted to ask you if you tried to run your <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> reproducer against it already ?</p></blockquote> <p>Yes. The <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> patch did appear to help with the evictions on unlink.</p> <p>Ok, found that 2.6.32-431.3.1 kernel is compatible with 2.4.2-14chaos kernel patches ...</p> <p>But now I encounter issues during the make step, Chris, do you use special tricks to build from your source tree ??</p> <p>James,<br/> It's too bad that at the time of this crash you ran with a Lustre debug mask made of only "D_IOCTL+D_NETERROR+D_WANING+D_ERROR+D_EMERG+D_HA+D_CONFIG+D_CONSOLE" and which did not contain at least "D_RPCTRACE+D_DLMTRACE" in addition ... That would have greatly helped to navigate in the crash-dump.</p> <p>Chris,<br/> Please, can you help+answer me about my previous request and detail any specific/procedure to build from LLNL source tree/git ?</p> <blockquote><p>But now I encounter issues during the make step, Chris, do you use special tricks to build from your source tree ??</p></blockquote> <p>Maybe? Could you be more specific about the commands you issued and the problem that you saw? If you had trouble in the liblustre part, try adding <tt>--disable-liblustre</tt> to the configure command line.</p> <p>I 1st cloned the chaos git, then selected the 2.4.2-14chaos tag.<br/> And then I applied the/our usual procedure described at "https://wiki.hpdd.intel.com/pages/viewpage.action?pageId=8126821", to generate a ldiskfs-based (sorry I forgot to mention this ...) Server build, and get these error during the "make rpms" step :</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ............... Wrote: /home/bruno.faccini/kernel/rpmbuild/SRPMS/lustre-iokit-1.4.0-1.src.rpm make[1]: Leaving directory `/home/bruno.faccini/lustre-chaos/lustre-iokit' rpmbuilddir=`mktemp -t -d lustre-build-$USER-XXXXXXXX`; \ make \ rpmbuilddir="$rpmbuilddir" \ rpm-local || exit 1; \ rpmbuild \ --define "_tmppath $rpmbuilddir/TMP" \ --define "_topdir $rpmbuilddir" \ --define "build_src_rpm 1" \ --define "dist %{nil}" \ -ts lustre-2.4.2.tar.gz || exit 1; \ cp $rpmbuilddir/SRPMS/*.src.rpm . || exit 1; \ rm -f -R $rpmbuilddir make[1]: Entering directory `/home/bruno.faccini/lustre-chaos' make[1]: Leaving directory `/home/bruno.faccini/lustre-chaos' error: Macro %rel_down has empty body error: Macro %rel_build has empty body error: line 96: Empty tag: Release: make: *** [srpm] Error 1 </pre> </div></div> <p>So I suspect I don't use the right procedure for your tree since we don't have such "rpmbuilddir" variable/macro in our build infrastructure, which plays with/changes "_topdir" setting ...</p> <p>Looks like a bug. You can work around it by using --with-downstream-release:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ sh autogen.sh $ ./configure --with-downstream-release=14chaos --disable-liblustre $ make rpms </pre> </div></div> <p>If you are trying to reproduce what we see, we use zfs not ldiskfs.</p> <p>Also, is reproduction likely to be an effective approach to this problem? What do we know about the code, and how do we intend to reproduce the issue?</p> <p>Ok. I think I got to the root of this (using my patch <a href="http://review.whamcloud.com/11842" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11842</a> to make every request to trigger a resend which is great for teting this code that is rarely hit in our testing, but gets hit quite a bit on larger systems now that LLNL added the patch to shrink client supplied buffers).</p> <p>First of all the code as is should have failed with this assert on the very first resent, but did not due to a bug. This was fixed in our tree, but not in 2.4 ( <a href="https://jira.whamcloud.com/browse/LU-3428" title="Fix &#39;uninitialized variables&#39; errors" class="issue-link" data-issue-key="LU-3428"><del>LU-3428</del></a>) <a href="http://review.whamcloud.com/6511" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/6511</a> - so you need this patch first.</p> <p>The assertion itself is due to a logic flaw. Then it becomes clear that patch <a href="https://jira.whamcloud.com/browse/LU-4584" title="Lock revocation process fails consistently" class="issue-link" data-issue-key="LU-4584"><del>LU-4584</del></a> you are carrying is wrong, in particular this part of it:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>--- a/lustre/mdt/mdt_open.c +++ b/lustre/mdt/mdt_open.c @@ -1710,6 +1710,10 @@ int mdt_reint_open(struct mdt_thread_info *info, struct m /* the open lock might already be gotten in * mdt_intent_fixup_resent */ LASSERT(lustre_msg_get_flags(req-&gt;rq_reqmsg) &amp; MSG_RESENT); + + /* take again a ref on old lock found to be resent */ + ldlm_lock_addref(&amp;lhc-&gt;mlh_reg_lh, lhc-&gt;mlh_reg_mode); + if (create_flags &amp; MDS_OPEN_LOCK) mdt_set_disposition(info, ldlm_rep, DISP_OPEN_LOCK); } else { </pre> </div></div> <p>The reason for that is because the mdt_intent_lock_replace assumes the lock has already been "replaced" into the client export, so it does not need any of those references - it cannot go away because it's already "owned" by the (not yet aware) client.</p> <p>With this part removed (and that other uninitialized var fix from above) I am no longer hitting the assertion or having terrible lock deadlocks on resend from the start.<br/> even despite that my racer testing on your tree cannot complete as other long fixed (in master tree) issues are getting in the way like <a href="https://jira.whamcloud.com/browse/LU-4725" title="wrong lock ordering in rename leads to deadlocks" class="issue-link" data-issue-key="LU-4725"><del>LU-4725</del></a> and <a href="https://jira.whamcloud.com/browse/LU-5144" title="rename vs link deadlock" class="issue-link" data-issue-key="LU-5144"><del>LU-5144</del></a></p> <p>At least this class of problems (<a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> related) should be extinguished for you now and hold you until you are ready to move to a newer release that has the more comprehensive lu-2827 patch with all of its afterfixes.</p> <p>I now refreshed <a href="http://review.whamcloud.com/#/c/9488/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9488/</a> to a more correct version.<br/> Also updated open_by_fid part to be aware of resends.</p> Related LU-4584 LU-3428 LU-5632 Development End date Mon, 10 Nov 2014 21:55:36 +0000 Rank 1|hzwu73: Rank (Obsolete) 15383 Severity Start date Wed, 20 Aug 2014 21:55:36 +0000