https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4740 Lustre <p>On our MDS, we seem to have a memory leak related to buffer cache that is<br/> unreclaimable. Our workload is extremely metadata intensive, so that the MDS is under constant heavy load.</p> <p>After a fresh reboot the buffer cache is filling up quickly. After a while RAM is used up and the machine starts swapping basically bringing Luster to a halt<br/> (clients disconnect, lock failures, etc.).</p> <p>The strange thing is that</p> <p>$ echo 3 &gt; /proc/sys/vm/drop_caches</p> <p>only frees part of the allocated buffer cache and after a while the unreclaimable part fills up RAM completely leading to the swap disaster.</p> <p>Setting /proc/sys/vm/vfs_cache_pressure &gt; 100 doesn't help and<br/> a large value of /proc/sys/vm/min_free_kbytes is happily ignored.</p> <p>Also strange: After unmounting all Lustre targets and even unloading the Lustre kernel modules the kernel still shows the amount of previously allocated buffer cache as used memory even though the amount of buffer cache is then shown as close to zero. So it seems we have big memory leak.</p> vanilla 2.6.32.61<br/> lustre 2.4.2<br/> Hardware: Dual Xeon L5640 / 24G RAM<br/> LU-4740

MDS - buffer cache not freed

Bug Critical Resolved Cannot Reproduce WC Triage Roland Fehrenbacher Sat, 8 Mar 2014 16:24:51 +0000 Sat, 9 Oct 2021 06:34:05 +0000 Sat, 9 Oct 2021 06:34:05 +0000 Lustre 2.4.2 0 8 <p>I have appended the output of /proc/meminfo and "slabtop -o -s c". Additionally, I've compiled in<br/> support for kmemleak, but "cat /sys/kernel/debug/kmemleak" is empty.</p> <p>According to meminfo swap is not used at all. Are you sure the logs are from the time when the issue comes up?</p> <p>No, these files were from about 1 hour after I started Lustre. <br/> To make things clearer, I deleted the old files and attached 3 pairs of meminfo and slabtop output. The first pair *.1<br/> is taken after a reboot before mounting Lustre. The second pair was taken 9 1/2 hours later right after a drop_caches and<br/> before I had to unmount Lustre and move things to the HA peer node (we don't want downtime during working hours).<br/> Note that the swapping deadlock stage hadn't been reached at this stage yet, but one can clearly see the RAM having disappeared:</p> <p>MemTotal - MemFree = MemUsed = 18GB<br/> Buffers + Cache + SLAB + misc = 14GB</p> <p>Also and most fatal buffer cache is not cleared by drop_caches.</p> <p>Finally the last pair is right after I unmounted Lustre. The buffer cache is cleared,<br/> but Active(file) remains at 12GB (pretty much the amount of unreclaimable buffer cache<br/> from before the umount).</p> <p>Roland, is it possible for you to test this with the current master branch? There have been a number of fixes related to memory usage on both the client and server.</p> <p>In particular, <a href="http://review.whamcloud.com/9223" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9223</a> fixed an allocation problem in 2.4 and 2.5, and there are others that are linked under <a href="https://jira.whamcloud.com/browse/LU-4053" title="client leaking objects/locks during IO" class="issue-link" data-issue-key="LU-4053"><del>LU-4053</del></a>.</p> <p>Andreas, I'll try. Can a 2.5.56 (git master) MDS/MDT work with 2.4.2 OSS/OSTs or do I need to update<br/> the whole cluster? Also note that on the client side, we're running a patched in-kernel client from 3.14-rc. Any expected compatibility problems with this combo? </p> <p>The 2.5 MDS is tested with 2.4 clients. We haven't tested with 3.14 clients, but I expect those to be similar to 2.4.x.</p> <p>Andreas,</p> <p>the issue is still there with 2.5.1 (where <a href="http://review.whamcloud.com/9223" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9223</a> is included) on the servers.</p> <p><a href="http://review.whamcloud.com/#/c/7942/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/7942/</a> which is also referenced under <a href="https://jira.whamcloud.com/browse/LU-4053" title="client leaking objects/locks during IO" class="issue-link" data-issue-key="LU-4053"><del>LU-4053</del></a> is not in 2.5.1, in fact is not retrievable<br/> in a cloned git repo. Some questions:</p> <ul class="alternate" type="square"> <li>Why is it not retrievable in a local clone even though it's visible on the git web interface ( <a href="http://git.whamcloud.com/?p=fs/lustre-release.git;a=commitdiff;h=99bdfc4c8e87d7a6038cb4856281302ecfe8ad34" class="external-link" target="_blank" rel="nofollow noopener">http://git.whamcloud.com/?p=fs/lustre-release.git;a=commitdiff;h=99bdfc4c8e87d7a6038cb4856281302ecfe8ad34</a> )?</li> </ul> <ul class="alternate" type="square"> <li>What's the status of this patch? Is it advisable to apply it?</li> </ul> <ul class="alternate" type="square"> <li>Do you have any other suggestions? We still need to reboot servers every couple of days.</li> </ul> <p>I would say that with the two -1 reviews on the 7942 patch should NOT be used in production. If the memory is not listed in slabs or in /proc/sys/lnet/memused then it is not being allocated directly by Lustre (which would print a "memory leaked" message at unmount. If the memory is not allocated directly by Lustre then the Lustre memory debugging code would not be useful for debugging this. It might be a bug in the ldiskfs code but it is hard to know. </p> <p>Are there unusual uses of the filesystem (e.g. Many create delete cycles in large directories) or some other way that this could be reproduced easily? Have you tried running with a server kernel using the in-kernel memory leak checking? I don't know much of the details, but I think this can be compiled for RHEL kernels. </p> Related LU-4053 Development Rank 1|hzwh7r: Rank (Obsolete) 13030 Severity