https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5530 Lustre <p>While testing 2.5 servers for a possible upcoming test shot I hit this bug while running simul.</p> <p>&lt;6&gt;[ 3724.190222] mdt00_003 D 0000000000000010 0 15957 2 0x00000000<br/> &lt;4&gt;[ 3724.197292] ffff8807e1f118b8 0000000000000046 0000000000000000 ffffffffa0f5b6eb<br/> &lt;4&gt;[ 3724.205018] ffff8810021cf190 ffff8810021cf138 ffff8808326a0538 ffffffffa0f5b6eb<br/> &lt;4&gt;[ 3724.212759] ffff8807f846fab8 ffff8807e1f11fd8 000000000000fbc8 ffff8807f846fab8<br/> &lt;4&gt;[ 3724.220491] Call Trace:<br/> &lt;4&gt;[ 3724.223032] <span class="error">&#91;&lt;ffffffff8152a6d5&gt;&#93;</span> rwsem_down_failed_common+0x95/0x1d0<br/> &lt;4&gt;[ 3724.229597] <span class="error">&#91;&lt;ffffffffa0d9d3fb&gt;&#93;</span> ? ldiskfs_xattr_trusted_get+0x2b/0x30 <span class="error">&#91;ldiskfs&#93;</span><br/> &lt;4&gt;[ 3724.237247] <span class="error">&#91;&lt;ffffffff811ae017&gt;&#93;</span> ? generic_getxattr+0x87/0x90<br/> &lt;4&gt;[ 3724.243199] <span class="error">&#91;&lt;ffffffff8152a866&gt;&#93;</span> rwsem_down_read_failed+0x26/0x30<br/> &lt;4&gt;[ 3724.249499] <span class="error">&#91;&lt;ffffffffa0fe8083&gt;&#93;</span> ? lod_xattr_get+0x153/0x420 <span class="error">&#91;lod&#93;</span><br/> &lt;4&gt;[ 3724.255867] <span class="error">&#91;&lt;ffffffff8128eab4&gt;&#93;</span> call_rwsem_down_read_failed+0x14/0x30<br/> &lt;4&gt;[ 3724.262580] <span class="error">&#91;&lt;ffffffff81529d64&gt;&#93;</span> ? down_read+0x24/0x30<br/> &lt;4&gt;[ 3724.267923] <span class="error">&#91;&lt;ffffffffa0f2569d&gt;&#93;</span> mdt_object_open_lock+0x1ed/0x9d0 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.274736] <span class="error">&#91;&lt;ffffffffa0f077e0&gt;&#93;</span> ? mdt_attr_get_complex+0x520/0x7f0 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.281720] <span class="error">&#91;&lt;ffffffffa0f2dcc7&gt;&#93;</span> mdt_reint_open+0x15b7/0x2150 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.288187] <span class="error">&#91;&lt;ffffffffa05e9f76&gt;&#93;</span> ? upcall_cache_get_entry+0x296/0x880 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt;[ 3724.295688] <span class="error">&#91;&lt;ffffffffa073fc10&gt;&#93;</span> ? lu_ucred+0x20/0x30 <span class="error">&#91;obdclass&#93;</span><br/> &lt;4&gt;[ 3724.301900] <span class="error">&#91;&lt;ffffffffa0f16611&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.307916] <span class="error">&#91;&lt;ffffffffa0efbe63&gt;&#93;</span> mdt_reint_internal+0x4c3/0x780 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.314551] <span class="error">&#91;&lt;ffffffffa0efc3ee&gt;&#93;</span> mdt_intent_reint+0x1ee/0x520 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.321023] <span class="error">&#91;&lt;ffffffffa0ef9bce&gt;&#93;</span> mdt_intent_policy+0x3ae/0x770 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.327619] <span class="error">&#91;&lt;ffffffffa085a2e5&gt;&#93;</span> ldlm_lock_enqueue+0x135/0x950 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 3724.334445] <span class="error">&#91;&lt;ffffffffa0883ccf&gt;&#93;</span><br/> ldlm_handle_enqueue0+0x50f/0x10c0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 3724.334445] <span class="error">&#91;&lt;ffffffffa0883ccf&gt;&#93;</span> ldlm_handle_enqueue0+0x50f/0x10c0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 3724.341608] <span class="error">&#91;&lt;ffffffffa0efa096&gt;&#93;</span> mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.347484] <span class="error">&#91;&lt;ffffffffa0efec5a&gt;&#93;</span> mdt_handle_common+0x52a/0x1470 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.354147] <span class="error">&#91;&lt;ffffffffa0f3b945&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> &lt;4&gt;[ 3724.360670] <span class="error">&#91;&lt;ffffffffa08b4015&gt;&#93;</span> ptlrpc_server_handle_request+0x385/0xc00 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 3724.368508] <span class="error">&#91;&lt;ffffffffa05ce4ce&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt;[ 3724.374864] <span class="error">&#91;&lt;ffffffffa05df3cf&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> &lt;4&gt;[ 3724.381791] <span class="error">&#91;&lt;ffffffffa08ab699&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 3724.388691] <span class="error">&#91;&lt;ffffffff810546b9&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> &lt;4&gt;[ 3724.394666] <span class="error">&#91;&lt;ffffffffa08b537d&gt;&#93;</span> ptlrpc_main+0xaed/0x1920 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 3724.402340] <span class="error">&#91;&lt;ffffffffa08b4890&gt;&#93;</span> ? ptlrpc_main+0x0/0x1920 <span class="error">&#91;ptlrpc&#93;</span><br/> &lt;4&gt;[ 3724.408702] <span class="error">&#91;&lt;ffffffff8109ab56&gt;&#93;</span> kthread+0x96/0xa0<br/> &lt;4&gt;[ 3724.413684] <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> &lt;4&gt;[ 3724.418754] <span class="error">&#91;&lt;ffffffff8109aac0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> &lt;4&gt;[ 3724.423841] <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20</p> RHEL6.5 MDS server running latest 2.5 with patches for<br/> <br/> <a href="https://jira.whamcloud.com/browse/LU-793" title="Reconnections should not be refused when there is a request in progress from this client." class="issue-link" data-issue-key="LU-793"><strike>LU-793</strike></a><br/> <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><strike>LU-2827</strike></a><br/> <a href="https://jira.whamcloud.com/browse/LU-3338" title="IOC_MDC_GETFILESTRIPE can abuse vmalloc()" class="issue-link" data-issue-key="LU-3338"><strike>LU-3338</strike></a><br/> <a href="https://jira.whamcloud.com/browse/LU-4933" title="Automatically tune the max_dirty_mb" class="issue-link" data-issue-key="LU-4933"><strike>LU-4933</strike></a><br/> <a href="https://jira.whamcloud.com/browse/LU-5266" title="LBUG on Failover -ldlm_process_extent_lock()) ASSERTION( lock-&gt;l_granted_mode != lock-&gt;l_req_mode ) " class="issue-link" data-issue-key="LU-5266"><strike>LU-5266</strike></a><br/> <a href="https://jira.whamcloud.com/browse/LU-5496" title="fix for LU-5266" class="issue-link" data-issue-key="LU-5496"><strike>LU-5496</strike></a><br/> LU-5530

MDS thread lockup witrh patched 2.5 server

Bug Major Resolved Fixed Oleg Drokin James A Simmons Thu, 21 Aug 2014 18:58:52 +0000 Wed, 13 Oct 2021 03:05:46 +0000 Sat, 11 Oct 2014 04:11:11 +0000 Lustre 2.5.2 Lustre 2.7.0 Lustre 2.5.4 0 11 <p>I uploaded a crash dump to ftp.whamcloud.com/uploads/<a href="https://jira.whamcloud.com/browse/LU-5530" title="MDS thread lockup witrh patched 2.5 server" class="issue-link" data-issue-key="LU-5530"><del>LU-5530</del></a></p> <p>Also you will see the debuginfo rpms as well.</p> <p>Niu</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi, James</p> <p>I downloaded the vmcore-dmesg.txt, it looks lots service threads are blocked on the inode-&gt;xattr_sem when trying to read xattr. unfortunately, downloading those files are extremely slow for me, it could take more than 10 hours to download a core file, I tried few times, and gave up at the end.</p> <p>Could you send me the output of the "bt -a" and "log" (two crash commands)? Thanks in advance.</p> <p>I did get the bt-a and log data but it didn't reveal much. The bt-a looked like the dmesg output. Perhaps we should approach this with kernel lockdep?</p> <p>Is it possible that the messages in vmcore-dmesg.txt was truncated? I think the output of bt-a and log could be still valuable. I never used lockdep, so I'm afraid that I can't give any advice on this.</p> <p>crash&gt; bt ffff8807e2630080<br/> PID: 15771 TASK: ffff8807e2630080 CPU: 20 COMMAND: "mdt01_000"<br/> #0 <span class="error">&#91;ffff8807e26817f8&#93;</span> schedule at ffffffff81528020<br/> #1 <span class="error">&#91;ffff8807e26818c0&#93;</span> rwsem_down_failed_common at ffffffff8152a6d5<br/> #2 <span class="error">&#91;ffff8807e2681920&#93;</span> rwsem_down_read_failed at ffffffff8152a866<br/> #3 <span class="error">&#91;ffff8807e2681960&#93;</span> call_rwsem_down_read_failed at ffffffff8128eab4<br/> #4 <span class="error">&#91;ffff8807e26819c8&#93;</span> mdt_object_open_lock at ffffffffa0f2569d <span class="error">&#91;mdt&#93;</span><br/> #5 <span class="error">&#91;ffff8807e2681a38&#93;</span> mdt_reint_open at ffffffffa0f2dcc7 <span class="error">&#91;mdt&#93;</span><br/> #6 <span class="error">&#91;ffff8807e2681b18&#93;</span> mdt_reint_rec at ffffffffa0f16611 <span class="error">&#91;mdt&#93;</span><br/> #7 <span class="error">&#91;ffff8807e2681b38&#93;</span> mdt_reint_internal at ffffffffa0efbe63 <span class="error">&#91;mdt&#93;</span><br/> #8 <span class="error">&#91;ffff8807e2681b78&#93;</span> mdt_intent_reint at ffffffffa0efc3ee <span class="error">&#91;mdt&#93;</span><br/> #9 <span class="error">&#91;ffff8807e2681bc8&#93;</span> mdt_intent_policy at ffffffffa0ef9bce <span class="error">&#91;mdt&#93;</span><br/> #10 <span class="error">&#91;ffff8807e2681c08&#93;</span> ldlm_lock_enqueue at ffffffffa085a2e5 <span class="error">&#91;ptlrpc&#93;</span><br/> #11 <span class="error">&#91;ffff8807e2681c78&#93;</span> ldlm_handle_enqueue0 at ffffffffa0883ccf <span class="error">&#91;ptlrpc&#93;</span><br/> #12 <span class="error">&#91;ffff8807e2681ce8&#93;</span> mdt_enqueue at ffffffffa0efa096 <span class="error">&#91;mdt&#93;</span><br/> #13 <span class="error">&#91;ffff8807e2681d08&#93;</span> mdt_handle_common at ffffffffa0efec5a <span class="error">&#91;mdt&#93;</span><br/> #14 <span class="error">&#91;ffff8807e2681d58&#93;</span> mds_regular_handle at ffffffffa0f3b945 <span class="error">&#91;mdt&#93;</span><br/> #15 <span class="error">&#91;ffff8807e2681d68&#93;</span> ptlrpc_server_handle_request at ffffffffa08b4015 <span class="error">&#91;ptlrpc&#93;</span><br/> #16 <span class="error">&#91;ffff8807e2681e48&#93;</span> ptlrpc_main at ffffffffa08b537d <span class="error">&#91;ptlrpc&#93;</span><br/> #17 <span class="error">&#91;ffff8807e2681ee8&#93;</span> kthread at ffffffff8109ab56<br/> #18 <span class="error">&#91;ffff8807e2681f48&#93;</span> kernel_thread at ffffffff8100c20a</p> <p>info</p> <p>[ 3724.166886] INFO: task mdt00_003:15957 blocked for more than 120 seconds.<br/> [ 3724.173775] Tainted: P --------------- 2.6.32-431.17.1.el6.wc.x86_64 #1<br/> [ 3724.182236] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message.<br/> [ 3724.190222] mdt00_003 D 0000000000000010 0 15957 2 0x00000000<br/> [ 3724.197292] ffff8807e1f118b8 0000000000000046 0000000000000000 ffffffffa0f5b6eb<br/> [ 3724.205018] ffff8810021cf190 ffff8810021cf138 ffff8808326a0538 ffffffffa0f5b6eb<br/> [ 3724.212759] ffff8807f846fab8 ffff8807e1f11fd8 000000000000fbc8 ffff8807f846fab8<br/> [ 3724.220491] Call Trace:<br/> [ 3724.223032] <span class="error">&#91;&lt;ffffffff8152a6d5&gt;&#93;</span> rwsem_down_failed_common+0x95/0x1d0<br/> [ 3724.229597] <span class="error">&#91;&lt;ffffffffa0d9d3fb&gt;&#93;</span> ? ldiskfs_xattr_trusted_get+0x2b/0x30 <span class="error">&#91;ldiskfs&#93;</span><br/> [ 3724.237247] <span class="error">&#91;&lt;ffffffff811ae017&gt;&#93;</span> ? generic_getxattr+0x87/0x90<br/> [ 3724.243199] <span class="error">&#91;&lt;ffffffff8152a866&gt;&#93;</span> rwsem_down_read_failed+0x26/0x30<br/> [ 3724.249499] <span class="error">&#91;&lt;ffffffffa0fe8083&gt;&#93;</span> ? lod_xattr_get+0x153/0x420 <span class="error">&#91;lod&#93;</span><br/> [ 3724.255867] <span class="error">&#91;&lt;ffffffff8128eab4&gt;&#93;</span> call_rwsem_down_read_failed+0x14/0x30<br/> [ 3724.262580] <span class="error">&#91;&lt;ffffffff81529d64&gt;&#93;</span> ? down_read+0x24/0x30<br/> [ 3724.267923] <span class="error">&#91;&lt;ffffffffa0f2569d&gt;&#93;</span> mdt_object_open_lock+0x1ed/0x9d0 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.274736] <span class="error">&#91;&lt;ffffffffa0f077e0&gt;&#93;</span> ? mdt_attr_get_complex+0x520/0x7f0 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.281720] <span class="error">&#91;&lt;ffffffffa0f2dcc7&gt;&#93;</span> mdt_reint_open+0x15b7/0x2150 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.288187] <span class="error">&#91;&lt;ffffffffa05e9f76&gt;&#93;</span> ? upcall_cache_get_entry+0x296/0x880 <span class="error">&#91;libcfs&#93;</span><br/> [ 3724.295688] <span class="error">&#91;&lt;ffffffffa073fc10&gt;&#93;</span> ? lu_ucred+0x20/0x30 <span class="error">&#91;obdclass&#93;</span><br/> [ 3724.301900] <span class="error">&#91;&lt;ffffffffa0f16611&gt;&#93;</span> mdt_reint_rec+0x41/0xe0 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.307916] <span class="error">&#91;&lt;ffffffffa0efbe63&gt;&#93;</span> mdt_reint_internal+0x4c3/0x780 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.314551] <span class="error">&#91;&lt;ffffffffa0efc3ee&gt;&#93;</span> mdt_intent_reint+0x1ee/0x520 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.321023] <span class="error">&#91;&lt;ffffffffa0ef9bce&gt;&#93;</span> mdt_intent_policy+0x3ae/0x770 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.327619] <span class="error">&#91;&lt;ffffffffa085a2e5&gt;&#93;</span> ldlm_lock_enqueue+0x135/0x950 <span class="error">&#91;ptlrpc&#93;</span></p> <p>[ 3724.334445] <span class="error">&#91;&lt;ffffffffa0883ccf&gt;&#93;</span> ldlm_handle_enqueue0+0x50f/0x10c0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 3724.341608] <span class="error">&#91;&lt;ffffffffa0efa096&gt;&#93;</span> mdt_enqueue+0x46/0xe0 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.347484] <span class="error">&#91;&lt;ffffffffa0efec5a&gt;&#93;</span> mdt_handle_common+0x52a/0x1470 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.354147] <span class="error">&#91;&lt;ffffffffa0f3b945&gt;&#93;</span> mds_regular_handle+0x15/0x20 <span class="error">&#91;mdt&#93;</span><br/> [ 3724.360670] <span class="error">&#91;&lt;ffffffffa08b4015&gt;&#93;</span> ptlrpc_server_handle_request+0x385/0xc00 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 3724.368508] <span class="error">&#91;&lt;ffffffffa05ce4ce&gt;&#93;</span> ? cfs_timer_arm+0xe/0x10 <span class="error">&#91;libcfs&#93;</span><br/> [ 3724.374864] <span class="error">&#91;&lt;ffffffffa05df3cf&gt;&#93;</span> ? lc_watchdog_touch+0x6f/0x170 <span class="error">&#91;libcfs&#93;</span><br/> [ 3724.381791] <span class="error">&#91;&lt;ffffffffa08ab699&gt;&#93;</span> ? ptlrpc_wait_event+0xa9/0x2d0 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 3724.388691] <span class="error">&#91;&lt;ffffffff810546b9&gt;&#93;</span> ? __wake_up_common+0x59/0x90<br/> [ 3724.394666] <span class="error">&#91;&lt;ffffffffa08b537d&gt;&#93;</span> ptlrpc_main+0xaed/0x1920 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 3724.402340] <span class="error">&#91;&lt;ffffffffa08b4890&gt;&#93;</span> ? ptlrpc_main+0x0/0x1920 <span class="error">&#91;ptlrpc&#93;</span><br/> [ 3724.408702] <span class="error">&#91;&lt;ffffffff8109ab56&gt;&#93;</span> kthread+0x96/0xa0<br/> [ 3724.413684] <span class="error">&#91;&lt;ffffffff8100c20a&gt;&#93;</span> child_rip+0xa/0x20<br/> [ 3724.418754] <span class="error">&#91;&lt;ffffffff8109aac0&gt;&#93;</span> ? kthread+0x0/0xa0<br/> [ 3724.423841] <span class="error">&#91;&lt;ffffffff8100c200&gt;&#93;</span> ? child_rip+0x0/0x20</p> <p>So I have looked in the core file, checked all mds threads.<br/> There is no obvious culprit for the deadlock observed. Nothing sleeping or otherwise blocked with this lock held.<br/> So it appears that the semaphore is somehow leaked.<br/> I inspected all the callers and users of the open semaphore and there does not seem to be any possible way for the leakage.<br/> I double-checked using ORNL patched 2.5 tree and it seems to be the case there as well.</p> <p>As such I am switching mode now.<br/> The plan is to trigger resending of the client requests at all times due to too small of a buffer. Either by making the server to always send huge replies or by the client to always provide too little ones.</p> <p>I am getting some interesting crashes this way already, though possibly this is bit self-inflicted.<br/> This is also useful so that we can execute this code even on our small test clusters during normal operations.</p> <p>My "force frequent resends to facilitate lu2827-like issues" patch reallyworked as expected ( <a href="http://review.whamcloud.com/11842" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11842</a> ).</p> <p>Now I was able to reproduce this problem and found out that it is <a href="https://jira.whamcloud.com/browse/LU-2827" title="mdt_intent_fixup_resent() cannot find the proper lock in hash" class="issue-link" data-issue-key="LU-2827"><del>LU-2827</del></a> related too - mismatched release of a semaphore (so that it was released more than it was taken) in mdt_reint_open.<br/> Proposed patch for this issue: <a href="http://review.whamcloud.com/11841" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11841</a> (also applies to b2_5)</p> <p>In addition, after fixing this I also hit <a href="https://jira.whamcloud.com/browse/LU-5579" title="MDS crashed by &quot;mdt_check_resent_lock()) ASSERTION( lock != NULL ) failed&quot;" class="issue-link" data-issue-key="LU-5579"><del>LU-5579</del></a> while testing with racer. And <a href="https://jira.whamcloud.com/browse/LU-5496" title="fix for LU-5266" class="issue-link" data-issue-key="LU-5496"><del>LU-5496</del></a> is also a bugfix in a related area so both of those patches should be adopted too.</p> <p>Landed for 2.5.4 and 2.7</p> Related LU-2827 LU-5686 Development Rank 1|hzwu9j: Rank (Obsolete) 15395 Severity