[LU-5333] rm cause MDS to complain hung tasks and disconnecting clients

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.3
• Type: Bug
• Priority: Major
• Reporter: Haisong Cai (Inactive)
• Assignee: Niu Yawei (Inactive)
• Resolution: Fixed
• Labels: sdsc

Environment

Linux puma-mds-10-6.local 2.6.32-358.23.2.el6_lustre.x86_64 #1 SMP Thu Dec 19 19:57:45 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

• Severity: 2

Description

A client was running "rm" to remove a couple of million files when MDS system load shot to 30 and kernel trace dumping complaining hung tasks - See attached output from "dmesg"

I would think this is normal workload for a duo-westmere CPU / 24GB RAM bonded myricom 10Gbps system.

We have been seeing happening more frequently in 2.4.3 than when we were at 1.8.7.

Anything suggestion?

thanks,
Haisong

Issue Links

• is related to: LU-5726 MDS buffer not freed when deleting files
• is related to: LU-5503 MDS (2.4.2) are getting "Service thre...

Comments

[Peter Jones]

Niu

Could you please advise on this issue?

Thanks

Peter

[Oleg Drokin]

From the traces it looks like it's a combination of OOM and journal deadlock of some sort.

[Haisong Cai (Inactive)]

Hi Oleg,

Later that day, the MDS came to a point where local commands were hanging.
We rebooted the server and ran e2fsck. That fixed a bunch quota entries and couple of inodes.

It has been stable so far.

Haisong

[Haisong Cai (Inactive)]

We have another case where removing some several million files from filesystem caused MDS to dump stack traces and gradually hang. I will attache some stack traces and dmesg following this message.

[Haisong Cai (Inactive)]

Correction: server is running 2.4.2 not 2.4.3

[Niu Yawei (Inactive)]

I think this could be related with LU-5726, and LU-5503 looks another instance of such problem.

[Haisong Cai (Inactive)]

Hi Yawei,

LU-5726 indicates the issue is fixed in 2.7.0.
Could you comment on whether the fix can be back-ported into earlier versions, specifically 2.5.*?

thanks,
Haisong

[Peter Jones]

Haisong

To be clear LU-5726 is targeted to be fixed in the 2.7 release but is not fixed yet. Your interest in this issue will raise the priority on this work and Niu will look at the possibilities/options to backport a fix to 2.5.x as part of this effort.

Regards

Peter

[Niu Yawei (Inactive)]

b2_5 port: http://review.whamcloud.com/#/c/13464/