https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5333 Lustre <p>A client was running "rm" to remove a couple of million files when MDS system load shot to 30 and kernel trace dumping complaining hung tasks - See attached output from "dmesg"</p> <p>I would think this is normal workload for a duo-westmere CPU / 24GB RAM bonded myricom 10Gbps system.</p> <p>We have been seeing happening more frequently in 2.4.3 than when we were at 1.8.7.</p> <p>Anything suggestion?</p> <p>thanks,<br/> Haisong</p> Linux puma-mds-10-6.local 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux<br/> LU-5333

rm cause MDS to complain hung tasks and disconnecting clients

Bug Major Resolved Fixed Niu Yawei Haisong Cai sdsc Fri, 11 Jul 2014 20:10:57 +0000 Wed, 22 Jun 2016 08:55:39 +0000 Wed, 22 Jun 2016 08:55:39 +0000 Lustre 2.4.3 0 6 <p>Niu</p> <p>Could you please advise on this issue?</p> <p>Thanks</p> <p>Peter</p> <p>From the traces it looks like it's a combination of OOM and journal deadlock of some sort.</p> <p>Hi Oleg,</p> <p>Later that day, the MDS came to a point where local commands were hanging. <br/> We rebooted the server and ran e2fsck. That fixed a bunch quota entries and couple of inodes.</p> <p>It has been stable so far.</p> <p>Haisong</p> <p>We have another case where removing some several million files from filesystem caused MDS to dump stack traces and gradually hang. I will attache some stack traces and dmesg following this message. </p> <p>Correction: server is running 2.4.2 not 2.4.3</p> <p>I think this could be related with <a href="https://jira.whamcloud.com/browse/LU-5726" title="MDS buffer not freed when deleting files" class="issue-link" data-issue-key="LU-5726"><del>LU-5726</del></a>, and <a href="https://jira.whamcloud.com/browse/LU-5503" title="MDS (2.4.2) are getting &quot;Service thread ... inactive&quot; and file-system times out" class="issue-link" data-issue-key="LU-5503"><del>LU-5503</del></a> looks another instance of such problem.</p> <p>Hi Yawei,</p> <p><a href="https://jira.whamcloud.com/browse/LU-5726" title="MDS buffer not freed when deleting files" class="issue-link" data-issue-key="LU-5726"><del>LU-5726</del></a> indicates the issue is fixed in 2.7.0. <br/> Could you comment on whether the fix can be back-ported into earlier versions, specifically 2.5.*?</p> <p>thanks,<br/> Haisong</p> <p>Haisong</p> <p>To be clear <a href="https://jira.whamcloud.com/browse/LU-5726" title="MDS buffer not freed when deleting files" class="issue-link" data-issue-key="LU-5726"><del>LU-5726</del></a> is targeted to be fixed in the 2.7 release but is not fixed yet. Your interest in this issue will raise the priority on this work and Niu will look at the possibilities/options to backport a fix to 2.5.x as part of this effort.</p> <p>Regards</p> <p>Peter</p> <p>b2_5 port: <a href="http://review.whamcloud.com/#/c/13464/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/13464/</a></p> Related LU-5726 LU-5503 Development End date Fri, 6 Feb 2015 20:10:57 +0000 Rank 1|hzwr8v: Rank (Obsolete) 14881 Severity Start date Fri, 11 Jul 2014 20:10:57 +0000