https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5617 Lustre <p>We had MDS hanging this morning, to a point where signing on server was denied. We had to power cycling the server in order to regain access to it.</p> <p>I am uploading "/var/log/messages" and 2 kernel trace dumping.<br/> I would need you help in interpreting from these logs and let me know <br/> where is likely the problem, the Lustre, networking, client overloading, and etc.</p> <p>thanks </p> Linux monkey-mds-10-3.local 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux<br/> LU-5617

MDS hang and would like to know the cause

Question/Request Minor Resolved Duplicate Niu Yawei Haisong Cai Fri, 12 Sep 2014 16:46:53 +0000 Mon, 21 Nov 2016 03:34:10 +0000 Mon, 21 Nov 2016 03:34:10 +0000 Lustre 2.4.2 0 5 <p>It appears from the MDS log that something was happening prior to the start of the log you attached. Can we get the log from the MDS for the 24 hours prior to the first log? <br/> Were there any indications of network errors? </p> <p>Hi Cliff,</p> <p>I am including /var/log/messages for MDS since last log rotation (Sept 7) here.<br/> For both Sept 10 and Sept 11, there were nothing logged.</p> <p>Niu</p> <p>Is there anything that you can determine from the information provided?</p> <p>Peter</p> <p>I saw lots of page allocation failures, looks your system is running short of memory, the reason is unclear to me, but it can be alleviated by tuning the vm parameters:</p> <ul class="alternate" type="square"> <li>Increasing the vm.min_free_kbytes;</li> <li>Set the vm.zone_reclaim_mode to 1;</li> </ul> <p>Hi Niu,</p> <p>Can you recommend a value to set for vm.min_free_kbytes? <br/> Our MDS has 24GB RAM.</p> <p>thanks,<br/> Haisong</p> <blockquote> <p>Can you recommend a value to set for vm.min_free_kbytes? <br/> Our MDS has 24GB RAM.</p></blockquote> <p>What's the current value? I don't have experience on tuning these values, I think you need to try some bigger value and see how it works. (but don't set it too large, probably less than 5% of total memory?)</p> <p>It because the system was running out of memory, it could be caused by <a href="https://jira.whamcloud.com/browse/LU-5726" title="MDS buffer not freed when deleting files" class="issue-link" data-issue-key="LU-5726"><del>LU-5726</del></a>. Dup of <a href="https://jira.whamcloud.com/browse/LU-5726" title="MDS buffer not freed when deleting files" class="issue-link" data-issue-key="LU-5726"><del>LU-5726</del></a>.</p> Development End date Mon, 15 Sep 2014 16:46:53 +0000 Rank 1|hzww5b: Rank (Obsolete) 15712 Start date Fri, 12 Sep 2014 16:46:53 +0000