https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4309 Lustre <p>An MDT thread hit an assertion in mds_intent_policy in what otherwise appeared to be normal operation.</p> <p>I'm attaching the kernel log messages after the LBUG. These are from the console. We have a crash dump from the node, but no lustre log files.</p> <p>Lustre build:<br/> Nov 18 12:46:55 widow-mds2 kernel: [ 387.597792] Lustre: Build Version: v1_8_9_WC1--CHANGED-2.6.18-348.3.1.el5.widow</p> RHEL 5.9/distro IB LU-4309

mds_intent_policy ASSERTION(new_lock != NULL) failed: op 0x8 lockh 0x0

Bug Critical Resolved Fixed Lai Siyao Blake Caldwell Mon, 25 Nov 2013 20:52:02 +0000 Mon, 10 Feb 2014 17:28:42 +0000 Mon, 10 Feb 2014 17:28:42 +0000 Lustre 1.8.9 0 6 <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Nov 18 11:46:10 widow-mds2 kernel: [1726316.746981] LustreError: dumping log to /tmp/lustre-log.1384793170.9088 </pre> </div></div> <p>Any chance to access /tmp/lustre-log.1384793170.9088?</p> <p>Hongchao</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Unfortunately, no logs from /tmp/ are left (on ramdisk).</p> <p>Lai</p> <p>I have realized that Hongchao is on vacation so could you please handle this one instead?</p> <p>Thanks</p> <p>Peter</p> <p>Could you know which client getattr cause this ASSERT? If so, can you check the backtrace of the process on the client that is doing getattr?</p> <p>Without the lustre logs in /tmp, I won't be able to track down the client. Even if the client could be identified from the crash dump, then there is the problem of identifying what it was doing at the time.</p> <p>I see that the dmesg output is not very helpful, but that's all I have other than a crash dump.</p> <p>So that we are better prepared for these cases in the future, what information can be collected on the server side beside /tmp/lustre.*? Collecting client debug logs is very difficult due to the number of clients. Would a ldlm_namespace_dump be helpful? If the LBUG has already occurred are there any debug flags for /proc/sys/lnet/debug that would provide useful information? Since the offending request has already been made, does capturing +net +dlmtrace +rpctrace do any good?</p> <p>Hmm, there is not much we can do in this case IMO, since MDS crash will cause all system hang, and it's hard to trace back to the client. I'll do more reviews on related code to understand this assert.</p> <p>I am not able to find the problem in the code, and I composed a debug patch to dump request before this assert, could you apply it and to get more info upon this failure again?</p> <p>Blake, </p> <p>Are you still seeing this assertion on your systems? If so, were you able to apply the patch to collect more information?</p> <p>Thanks, <br/> James</p> <p>I haven't been able to apply this debug patch yet. The system has been stable, and as a result we haven't had an unschedule outage to apply that patch. So nothing at this time. I will try applying the debug patch to another system that we can take down sooner.</p> <p>Blake, Thanks for the update.</p> <p>So this filesystem is out of production (in a hold state before decommissioning); my assertion is that we should go ahead and close this issue &#8211; even if we integrated the patch and ran the storage system with it for a while it would never get any client access and likely would not exercise the code path for the patch. Any objections?</p> <p>&#8211;<br/> -Jason</p> <p>Let's close it.</p> <p>Thank you for the update. I will close this ticket.</p> Development Rank 1|hzwa27: Rank (Obsolete) 11801 Severity