[LU-4740] MDS - buffer cache not freed

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.4.2
• Type: Bug
• Priority: Critical
• Reporter: Roland Fehrenbacher
• Assignee: WC Triage
• Resolution: Cannot Reproduce

Environment

vanilla 2.6.32.61
lustre 2.4.2
Hardware: Dual Xeon L5640 / 24G RAM

• Severity: 4

Description

On our MDS, we seem to have a memory leak related to buffer cache that is
unreclaimable. Our workload is extremely metadata intensive, so that the MDS is under constant heavy load.

After a fresh reboot the buffer cache is filling up quickly. After a while RAM is used up and the machine starts swapping basically bringing Luster to a halt
(clients disconnect, lock failures, etc.).

The strange thing is that

$ echo 3 > /proc/sys/vm/drop_caches

only frees part of the allocated buffer cache and after a while the unreclaimable part fills up RAM completely leading to the swap disaster.

Setting /proc/sys/vm/vfs_cache_pressure > 100 doesn't help and
a large value of /proc/sys/vm/min_free_kbytes is happily ignored.

Also strange: After unmounting all Lustre targets and even unloading the Lustre kernel modules the kernel still shows the amount of previously allocated buffer cache as used memory even though the amount of buffer cache is then shown as close to zero. So it seems we have big memory leak.

Issue Links

• is related to: LU-4053 client leaking objects/locks during IO

Comments

[Roland Fehrenbacher]

I have appended the output of /proc/meminfo and "slabtop -o -s c". Additionally, I've compiled in
support for kmemleak, but "cat /sys/kernel/debug/kmemleak" is empty.

[Bernd Schubert]

According to meminfo swap is not used at all. Are you sure the logs are from the time when the issue comes up?

[Roland Fehrenbacher]

No, these files were from about 1 hour after I started Lustre.
To make things clearer, I deleted the old files and attached 3 pairs of meminfo and slabtop output. The first pair *.1
is taken after a reboot before mounting Lustre. The second pair was taken 9 1/2 hours later right after a drop_caches and
before I had to unmount Lustre and move things to the HA peer node (we don't want downtime during working hours).
Note that the swapping deadlock stage hadn't been reached at this stage yet, but one can clearly see the RAM having disappeared:

MemTotal - MemFree = MemUsed = 18GB
Buffers + Cache + SLAB + misc = 14GB

Also and most fatal buffer cache is not cleared by drop_caches.

Finally the last pair is right after I unmounted Lustre. The buffer cache is cleared,
but Active(file) remains at 12GB (pretty much the amount of unreclaimable buffer cache
from before the umount).

[Andreas Dilger]

Roland, is it possible for you to test this with the current master branch? There have been a number of fixes related to memory usage on both the client and server.

In particular, http://review.whamcloud.com/9223 fixed an allocation problem in 2.4 and 2.5, and there are others that are linked under LU-4053.

[Roland Fehrenbacher]

Andreas, I'll try. Can a 2.5.56 (git master) MDS/MDT work with 2.4.2 OSS/OSTs or do I need to update
the whole cluster? Also note that on the client side, we're running a patched in-kernel client from 3.14-rc. Any expected compatibility problems with this combo?

[Andreas Dilger]

The 2.5 MDS is tested with 2.4 clients. We haven't tested with 3.14 clients, but I expect those to be similar to 2.4.x.

[Roland Fehrenbacher]

Andreas,

the issue is still there with 2.5.1 (where http://review.whamcloud.com/9223 is included) on the servers.

http://review.whamcloud.com/#/c/7942/ which is also referenced under LU-4053 is not in 2.5.1, in fact is not retrievable
in a cloned git repo. Some questions:

• Why is it not retrievable in a local clone even though it's visible on the git web interface ( http://git.whamcloud.com/?p=fs/lustre-release.git;a=commitdiff;h=99bdfc4c8e87d7a6038cb4856281302ecfe8ad34 )?

• What's the status of this patch? Is it advisable to apply it?

• Do you have any other suggestions? We still need to reboot servers every couple of days.

[Andreas Dilger]

I would say that with the two -1 reviews on the 7942 patch should NOT be used in production. If the memory is not listed in slabs or in /proc/sys/lnet/memused then it is not being allocated directly by Lustre (which would print a "memory leaked" message at unmount. If the memory is not allocated directly by Lustre then the Lustre memory debugging code would not be useful for debugging this. It might be a bug in the ldiskfs code but it is hard to know.

Are there unusual uses of the filesystem (e.g. Many create delete cycles in large directories) or some other way that this could be reproduced easily? Have you tried running with a server kernel using the in-kernel memory leak checking? I don't know much of the details, but I think this can be compiled for RHEL kernels.