We have had some problems in recent weeks with the MDS on grove (sequoia's filesystem cluster) thrashing for anywhere from minutes to many hours while under load. While it does so, it is not appear to be handling traffic very quickly, and the node load is so high that login is nearly impossible.

I caught it doing that for a while today during testing and dumped some SysRq info to the console.

It looks to me like the active tasks may be spending too much time under ptlrpc_alloc_rqbd() doing vmallocs.

Prakash had a patch to move those allocations to a slab. But it became time consuming to keep moving forward. We may need to look at reviving that.

See attached file "console.grove-mds1.txt.bz2".