[LU-5617] MDS hang and would like to know the cause

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.2
• Type: Question/Request
• Priority: Minor
• Reporter: Haisong Cai (Inactive)
• Assignee: Niu Yawei (Inactive)
• Resolution: Duplicate

Environment

Linux monkey-mds-10-3.local 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

Description

We had MDS hanging this morning, to a point where signing on server was denied. We had to power cycling the server in order to regain access to it.

I am uploading "/var/log/messages" and 2 kernel trace dumping.
I would need you help in interpreting from these logs and let me know
where is likely the problem, the Lustre, networking, client overloading, and etc.

thanks

Comments

[Cliff White (Inactive)]

It appears from the MDS log that something was happening prior to the start of the log you attached. Can we get the log from the MDS for the 24 hours prior to the first log?
Were there any indications of network errors?

[Haisong Cai (Inactive)]

Hi Cliff,

I am including /var/log/messages for MDS since last log rotation (Sept 7) here.
For both Sept 10 and Sept 11, there were nothing logged.

[Peter Jones]

Niu

Is there anything that you can determine from the information provided?

Peter

[Niu Yawei (Inactive)]

I saw lots of page allocation failures, looks your system is running short of memory, the reason is unclear to me, but it can be alleviated by tuning the vm parameters:

• Increasing the vm.min_free_kbytes;
• Set the vm.zone_reclaim_mode to 1;

[Haisong Cai (Inactive)]

Hi Niu,

Can you recommend a value to set for vm.min_free_kbytes?
Our MDS has 24GB RAM.

thanks,
Haisong

[Niu Yawei (Inactive)]

Can you recommend a value to set for vm.min_free_kbytes?
Our MDS has 24GB RAM.

What's the current value? I don't have experience on tuning these values, I think you need to try some bigger value and see how it works. (but don't set it too large, probably less than 5% of total memory?)

[Niu Yawei (Inactive)]

It because the system was running out of memory, it could be caused by LU-5726. Dup of LU-5726.