https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2708 Lustre <p>We have had some problems in recent weeks with the MDS on grove (sequoia's filesystem cluster) thrashing for anywhere from minutes to many hours while under load. While it does so, it is not appear to be handling traffic very quickly, and the node load is so high that login is nearly impossible.</p> <p>I caught it doing that for a while today during testing and dumped some SysRq info to the console.</p> <p>It looks to me like the active tasks may be spending too much time under ptlrpc_alloc_rqbd() doing vmallocs.</p> <p>Prakash had a patch to move those allocations to a slab. But it became time consuming to keep moving forward. We may need to look at reviving that.</p> <p>See attached file "console.grove-mds1.txt.bz2".</p> Lustre 2.3.58-6chaos (github.com/chaos/lustre) on MDS. LU-2708

MDS thrashing in ptlrpc_alloc_rqbd

Bug Blocker Resolved Fixed Liang Zhen Christopher Morrone HB sequoia topsequoia Tue, 29 Jan 2013 20:48:58 +0000 Wed, 6 Mar 2013 20:42:07 +0000 Wed, 6 Mar 2013 20:42:07 +0000 Lustre 2.4.0 0 5 <p>Related to <a href="https://jira.whamcloud.com/browse/LU-2432" title="ptlrpc_alloc_rqbd spinning on vmap_area_lock on MDS" class="issue-link" data-issue-key="LU-2432"><del>LU-2432</del></a>. However, I verified that we are running with the kernel patch that fixes the kernel vmalloc problem for the problem noted in this ticket.</p> <p>I have posted another patch : <a href="http://review.whamcloud.com/#change,4940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4940</a><br/> it will not fundamentally resolve the issue, but it will largely reduce request buffer size/count, and save a lot of memory. <br/> But I think it has to be reviewed by Andreas first.</p> <p>we have landed two patches for this:<br/> <a href="http://review.whamcloud.com/#change,4939" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4939</a> can largely decrease threads number contending on vmalloc.<br/> <a href="http://review.whamcloud.com/#change,4940" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,4940</a> can improve buffer utilization rate, and decrease chance to call vmalloc.<br/> So I think we can close this ticket for now.</p> Related LU-2432 Development Rank 1|hzvgg7: Rank (Obsolete) 6314