[LU-4579] Timeout system horribly broken

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.6.0, Lustre 2.5.3
• Environment: Lustre 2.4.0-21chaos
• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Christopher Morrone (Inactive)

Description

It would appear that the timeout system in Lustre is horribly broken in Lustre 2.4.0-21chaos (see github.com/chaos/lustre). On MDS nodes, we frequently see problems where almost all of the mdt threads are stuck waiting under in ldlm_completion_ast(). We see warning on the console from the kernel that these threads are sleeping for in excess of 1200 seconds, despite an at_max of 600 seconds.

The problems get worse than that, sometimes we'll see clients evicted by an mdt after 9000+ seconds. Obviously, that isn't acceptable.

The practical effect of these poorly handled timeouts are file systems that go unresponsive for hours (if not days) at a time.

We need to work out a plan to fix the timeouts in lustre.

Issue Links

• is related to: LU-4942 lock callback timeout is not per-export
• is related to: LU-5497 Many MDS service threads blocked in ldlm_completion_ast()
• is related to: LU-4786 Apparent denial of service from client to mdt
• is related to: LU-4570 Metadata slowdowns on production filesystem at ORNL