https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4148 Lustre <p>Running mdtest, seeing a performance drop in rmdir. <br/> All clients appear to be hitting watchdogs, example:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">INFO: task mdtest:7072 blocked <span class="code-keyword">for</span> more than 120 seconds. <span class="code-quote">"echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs"</span> disables <span class="code-keyword">this</span> message. mdtest D 0000000000000009 0 7072 7058 0x00000000 ffff880870771e08 0000000000000082 ffff880871506aa0 ffff880871506aa0 ffff880871506aa0 000000000000000b ffff880871506aa0 0000001081065d54 ffff880871507058 ffff880870771fd8 000000000000fb88 ffff880871507058 Call Trace: [&lt;ffffffff8118f541&gt;] ? path_put+0x31/0x40 [&lt;ffffffff8150f78e&gt;] __mutex_lock_slowpath+0x13e/0x180 [&lt;ffffffff8150f62b&gt;] mutex_lock+0x2b/0x50 [&lt;ffffffff81192367&gt;] do_rmdir+0xb7/0x120 [&lt;ffffffff8100c535&gt;] ? math_state_restore+0x45/0x60 [&lt;ffffffff81192426&gt;] sys_rmdir+0x16/0x20 [&lt;ffffffff8100b072&gt;] system_call_fastpath+0x16/0x1b </pre> </div></div> <p>No errors on MDS</p> Hyperion/LLNL LU-4148

Clients experiencing massive watchdogs in mdtest rmdir

Bug Major Resolved Cannot Reproduce Lai Siyao Cliff White Fri, 25 Oct 2013 21:04:18 +0000 Sat, 9 Oct 2021 06:18:08 +0000 Sat, 9 Oct 2021 06:18:08 +0000 Lustre 2.5.0 0 6 <p>I guess in these cases it would be very helpful to have a list of all processes with stacktraces to see where does this happen, on client or if a server thread got wedged (less likely because of no errors on MDTs I guess).</p> <p>a crashdump from such a client might be helpful too. I assume no other errors in the client logs?</p> <p>At this point i am only seeing the watchdogs, will recreate again, maybe with fewer clients. The test does complete eventually, there are no errors causing test aborts.</p> <p>so, mdt is jsut slow apparently.<br/> Is this happening on share dir delete?</p> <p>Lai</p> <p>Oleg was wondering if this might be a wide-effect of this patch - <a href="http://review.whamcloud.com/7257" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7257</a> </p> <p>What do you think? If not, do you have some other idea?</p> <p>Thanks</p> <p>Peter</p> <p>The backtrace shows the process is waiting on parent i_mutex in do_rmdir(), if this can be reproduced and see which process is holding this lock, it can help analyse the cause.</p> <p><a href="http://review.whamcloud.com/7257" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/7257</a> doesn't look to be a direct cause of this slowness if there are no processes which changes parent directory permission constantly.</p> <p>Dunp of all registers and stacks from a hung client</p> Development Rank 1|hzw6wn: Rank (Obsolete) 11263 Severity