[LU-2708] MDS thrashing in ptlrpc_alloc_rqbd

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.0
• Type: Bug
• Priority: Blocker
• Reporter: Christopher Morrone
• Assignee: Liang Zhen (Inactive)
• Resolution: Fixed
• Labels: HB, sequoia, topsequoia

Environment

Lustre 2.3.58-6chaos (github.com/chaos/lustre) on MDS.

Description

We have had some problems in recent weeks with the MDS on grove (sequoia's filesystem cluster) thrashing for anywhere from minutes to many hours while under load. While it does so, it is not appear to be handling traffic very quickly, and the node load is so high that login is nearly impossible.

I caught it doing that for a while today during testing and dumped some SysRq info to the console.

It looks to me like the active tasks may be spending too much time under ptlrpc_alloc_rqbd() doing vmallocs.

Prakash had a patch to move those allocations to a slab. But it became time consuming to keep moving forward. We may need to look at reviving that.

See attached file "console.grove-mds1.txt.bz2".

Issue Links

• is related to: LU-2432 ptlrpc_alloc_rqbd spinning on vmap_ar...

Comments

[Christopher Morrone]

Related to LU-2432. However, I verified that we are running with the kernel patch that fixes the kernel vmalloc problem for the problem noted in this ticket.

[Liang Zhen (Inactive)]

I have posted another patch : http://review.whamcloud.com/#change,4940
it will not fundamentally resolve the issue, but it will largely reduce request buffer size/count, and save a lot of memory.
But I think it has to be reviewed by Andreas first.

[Liang Zhen (Inactive)]

we have landed two patches for this:
http://review.whamcloud.com/#change,4939 can largely decrease threads number contending on vmalloc.
http://review.whamcloud.com/#change,4940 can improve buffer utilization rate, and decrease chance to call vmalloc.
So I think we can close this ticket for now.