https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6189 Lustre <p>This morning within a few hours of each other, we hit this LBUG which caused the MDS to crash. The first time after reboot we had to abort recovery to get lustre back. We have a crashdump from the MDS.</p> <p>Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.805235] LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 375s: evicting client at 4966@gni100 ns: mdt-<br/> atlas1-MDT0000_UUID lock: ffff881ec6e16c80/0xfc6e8aed747d1af2 lrc: 4/0,0 mode: CR/CR res: <span class="error">&#91;0x2001a597a:0x85:0x0&#93;</span>.0 bits 0x2 rrc: 4 type: IBT flags: 0x60200000000020 nid: 4966@gni100 remote: 0x20ee476ee499c158<br/> expref: 132 pid: 16827 timeout: 4301930544 lvb_type: 0<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.858358] LustreError: 16827:0:(mdt_handler.c:4078:mdt_intent_reint()) ASSERTION( rc == 0 ) failed: Error occurred but lock handle is still in use, rc = -1<br/> 16<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.874660] LustreError: 16827:0:(mdt_handler.c:4078:mdt_intent_reint()) LBUG<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.882757] Pid: 16827, comm: mdt00_224<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.887151] <br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.887152] Call Trace:<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.891770] <span class="error">&#91;&lt;ffffffffa0407895&gt;&#93;</span> libcfs_debug_dumpstack+0x55/0x80 <span class="error">&#91;libcfs&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.899670] <span class="error">&#91;&lt;ffffffffa0407e97&gt;&#93;</span> lbug_with_loc+0x47/0xb0 <span class="error">&#91;libcfs&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.906710] <span class="error">&#91;&lt;ffffffffa0d4379a&gt;&#93;</span> mdt_intent_reint+0x51a/0x520 <span class="error">&#91;mdt&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.913933] <span class="error">&#91;&lt;ffffffffa0d40c4e&gt;&#93;</span> mdt_intent_policy+0x3ae/0x770 <span class="error">&#91;mdt&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.921281] <span class="error">&#91;&lt;ffffffffa06de2e5&gt;&#93;</span> ldlm_lock_enqueue+0x135/0x980 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.928910] <span class="error">&#91;&lt;ffffffffa0707d0b&gt;&#93;</span> ldlm_handle_enqueue0+0x51b/0x10c0 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.936903] <span class="error">&#91;&lt;ffffffff81069f75&gt;&#93;</span> ? enqueue_entity+0x125/0x450<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.943544] <span class="error">&#91;&lt;ffffffffa0d41116&gt;&#93;</span> mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.950094] <span class="error">&#91;&lt;ffffffffa0d4602a&gt;&#93;</span> mdt_handle_common+0x52a/0x1470 <span class="error">&#91;mdt&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.957515] <span class="error">&#91;&lt;ffffffffa0d833e5&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.964770] <span class="error">&#91;&lt;ffffffffa0737fe5&gt;&#93;</span> ptlrpc_server_handle_request+0x385/0xc00 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.973547] <span class="error">&#91;&lt;ffffffffa04084ce&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.980677] <span class="error">&#91;&lt;ffffffffa04193cf&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.988407] <span class="error">&#91;&lt;ffffffffa072f6c9&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7271.996116] <span class="error">&#91;&lt;ffffffff810546b9&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7272.002774] <span class="error">&#91;&lt;ffffffffa073934d&gt;&#93;</span> ptlrpc_main+0xaed/0x1760 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7272.009920] <span class="error">&#91;&lt;ffffffffa0738860&gt;&#93;</span> ? ptlrpc_main+0x0/0x1760 <span class="error">&#91;ptlrpc&#93;</span><br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7272.017040] <span class="error">&#91;&lt;ffffffff8109ab56&gt;&#93;</span> kthread+0x96/0xa0<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7272.022607] <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7272.028267] <span class="error">&#91;&lt;ffffffff8109aac0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7272.033930] <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20<br/> Feb 1 10:03:15 atlas-mds1.ccs.ornl.gov kernel: [ 7272.039782] </p> LU-6189

LustreError: (mdt_handler.c:4078:mdt_intent_reint()) ASSERTION( rc == 0 ) failed: Error occurred but lock handle is still in use, rc = -116

Bug Critical Resolved Duplicate Peter Jones Philip B Curtis Sun, 1 Feb 2015 16:23:10 +0000 Mon, 4 Jan 2016 18:20:29 +0000 Thu, 2 Apr 2015 17:23:55 +0000 Lustre 2.5.3 0 7 <p>Philip</p> <p>You have entered this ticket as a Severity 1 which means that the filesystem is down. Is this the case? From the description it sounds like service has been restored but you want to treat this as a high priority to prevent further such crashes.</p> <p>Peter</p> <p>Peter</p> <p>No, the first time this occurred lustre was restarted. I haven't brought lustre back up this time since this was following so closely to the first time. I wanted to get Intel involved before I attempted another start.</p> <p>Philip</p> <p>ok. I think that it is best to start uploading the crash dump to our ftp site in case that is useful. Do you have the instructions on how to do that? Also, is the code being run exactly in sync with the tip of your b2_5 branch on gut hub? <a href="https://github.com/ORNL-TechInt/lustre/commits/b2_5" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/ORNL-TechInt/lustre/commits/b2_5</a></p> <p>I'm quite sure this is fixed with <a href="http://review.whamcloud.com/#/c/12828/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/12828/</a></p> <p>Philip</p> <p>This is a patch that needs to be applied to the MDS only. Is there anything else that you need from us at this point before attempting to bring the filesystem back up?</p> <p>Peter</p> <p>No, I do not have instructions for the ftp site. That is correct, we are at the tip of the code there.</p> <p>We are running what is in the ORNL git hub. We attempted a upgrade but it failed after a few days. I general don't upgrade the ORNL branch for a few weeks after a upgrade just in case something goes wrong.</p> <p>Nope. I will get you those crashdumps once I have those instructions and I will see about getting this patched version in place and we will go from there.</p> <p>We have rebooted into the new RPMs with the patch. Lustre has started and I will continue to monitor. Thank you for your help.</p> <p>Philip</p> <p>Good news. Thanks for the update. I will drop the severity to S2 and will continue to monitor in case there are any further complications.</p> <p>As per ORNL ok to close as <a href="https://jira.whamcloud.com/browse/LU-5934" title="mdt_intent_reint()) ASSERTION( rc == 0 ) failed: Error occurred but lock handle is still in use, rc = -2" class="issue-link" data-issue-key="LU-5934"><del>LU-5934</del></a> has landed</p> Related LU-5934 Development Rank 1|hzx5dj: Rank (Obsolete) 17312 Severity