[LU-5525] ASSERTION( new_lock->l_readers + new_lock->l_writers == 0 ) failed

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Labels: llnl
• Environment: MDS node, Lustre 2.4.2-14chaos, ZFS OBD
• Severity: 3
• Assignee: Bruno Faccini (Inactive)
• Reporter: Christopher Morrone (Inactive)

Description

After upgrading to lustre 2.4.2-14chaos (see github.com/chaos/lustre), we soon hit the following assertion on one of our MDS nodes:

mdt_handler.c:3652:mdt_intent_lock_replace()) ASSERTION( new_lock->l_readers + new_lock->l_writers == 0 ) failed

Perhaps most significantly, this tag of our lustre tree includes the patch entitled:

LU-4584 mdt: ensure orig lock is found in hash upon resend

James Simmons reported this assertion when he tested the LU-4584 patch, but the Bruno made the evaluation that the assertion was unrelated to the patch.

Whether it is related or not, we need to fix the problem.

Issue Links

• is related to: LU-5632 ldlm_lock_addref()) ASSERTION( lock != ((void *)0) )
• is related to: LU-4584 Lock revocation process fails consistently
• is related to: LU-3428 Fix 'uninitialized variables' errors

Comments

[Bruno Faccini (Inactive)]

James,
It's too bad that at the time of this crash you ran with a Lustre debug mask made of only "D_IOCTL+D_NETERROR+D_WANING+D_ERROR+D_EMERG+D_HA+D_CONFIG+D_CONSOLE" and which did not contain at least "D_RPCTRACE+D_DLMTRACE" in addition ... That would have greatly helped to navigate in the crash-dump.

[Bruno Faccini (Inactive)]

But now I encounter issues during the make step, Chris, do you use special tricks to build from your source tree ??

[Bruno Faccini (Inactive)]

Ok, found that 2.6.32-431.3.1 kernel is compatible with 2.4.2-14chaos kernel patches ...

[Christopher Morrone (Inactive)]

I wanted to ask you if you tried to run your LU-4584 reproducer against it already ?

Yes. The LU-4584 patch did appear to help with the evictions on unlink.

[James A Simmons]

I can report I don't see this issue with 2.5 servers. I see another problem instead.

[Bruno Faccini (Inactive)]

Chris, can you help and tell me the kernel version you use/build with 2.4.2-14chaos, I am having problems when patching the 2.6.32-358.23.2 kernel version we use to build our latest b2_4/2.4.3-RC1 branch/tag.

[James A Simmons]

Oops, you are right. I saw this problem with our 2.4 image which did contain the LU-4584 patch. Sorry for the confusion. I pushed the proper debug rpms this time.

[Bruno Faccini (Inactive)]

Chris, I am really sorry about the fact my b2_4 patch for LU-4584 seems to cause you so many troubles.
This LBUG already has not occured when running auto-tests and your/LU-4584 reproducer, against Jenkins build for patch-set #7 of Gerrit change #9488, and I am still unable to reproduce it when running with a local build made of tag 2.4.3-RC1 + LU-4584 patch (ie, patch-set #7 of Gerrit change #9488).
I am currently building a Lustre version based on 2.4.2-14chaos and see if I can reproduce in-house, but I wanted to ask you if you tried to run your LU-4584 reproducer against it already ?

[Bruno Faccini (Inactive)]

James, "crash" tool claims "WARNING: kernel version inconsistency between vmlinux and dumpfile" and gives up due to further errors ...
Are you sure that at the time of crash you were running with this "2.6.32-431.17.1.el6.wc" kernel version for which you provided the debuginfo RPMs ? BTW, strings out from vmcore and vmcore-dmesg.txt report "2.6.32-358.23.2.el6.atlas"...

[James A Simmons]

Sorry I forgot the debug.rpms. I just uploaded them to the same ftp spot. ORNL would really like to go to 2.5 ASAP.

Yes, I built b2_5 at commit a43e0e4ce4b57240540e8d35a8ba44e203c70ae1 with some additional patches. We tag our kernels to avoid our automated rpm update system from stomping on them. They basically are the RHEL Lustre patched kernels. We also build our own kernel rpms to enable certain things like timestamps in the printk.

[Bruno Faccini (Inactive)]

Hello James, thanks for this crash-dump already! But I also need the corresponding vmlinux and Lustre modules to allow crash tool to run on it. By "the latest b2_5 branch with the LU-2827 patch", do you mean you used one of our recent Jenkins build to get this crash? If yes, can you point me to which one ? And if not, can you also xfer the 2x kernel-debuginfo[-common] RPMs and also the luste-modules (or the lustre-debuginfo) RPM ?
BTW, I checked the vmcore-dmesg.txt file, you provided with the crash-dump, and its content seems to indicate that you run with both a Kernel and Lustre distro you have builded locally, right?