[LU-5726] MDS buffer not freed when deleting files

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.7.0, Lustre 2.5.4
• Affects Version/s: Lustre 2.4.3
• Environment: CentOS 6.5; Kernel 2.6.32-358.23.2
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Rick Mohr

Description

When deleting large numbers of files, memory usage on the MDS server grows significantly. Attempts to reclaim memory by dropping caches only results in some of the memory being freed. The buffer usage continues to grow until eventually the MDS server starts OOMing.

The rate at which the buffer usage grows seems to vary but looks like it might be based on the number of clients that are deleting files and the speed at which the files are deleted.

Issue Links

• is related to: LU-5333 rm cause MDS to complain hung tasks and disconnecting clients
• is related to: LU-5841 Lustre 2.4.2 MDS, hitting OOM errors

Comments

[Peter Jones]

Great news. Landed for 2.5.4 and 2.7

[Rick Mohr]

Niu,

I checked the MDS mem usage again this morning:

MemTotal:       66053640 kB
MemFree:         5568288 kB
Buffers:        55504980 kB
Active:         22374284 kB
Inactive:       33260116 kB

After I dropped caches:

MemTotal:       66053640 kB
MemFree:        63146420 kB
Buffers:           59788 kB
Active:            57960 kB
Inactive:          93452 kB

Looks like the patch is successful.

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: http://review.whamcloud.com/13655
Subject: LU-5726 ldiskfs: missed brelse() in large EA patch
Project: fs/lustre-release
Branch: b2_4
Current Patch Set: 1
Commit: 8e29ef136f426fef66b6008d379afc5e5ddc4ab5

[Rick Mohr]

Niu,

We just applied the patch today to our production file system (Lustre 2.4.3) and are running some heavy purges right now. I collected some info about the memory usage. Prior to the patch, it seemed like the memory growth was dominated by the "Inactive(file)" in /proc/meminfo. I dropped the cache in the MDS server (echo 3 > /proc/sys/vm/drop_caches) and collected Inactive(file) usage every minute:

nactive(file): 1146656 kB
Inactive(file): 3426128 kB
Inactive(file): 5510484 kB
Inactive(file): 6634728 kB
Inactive(file): 7514500 kB
Inactive(file): 8075948 kB
Inactive(file): 8662528 kB
Inactive(file): 9210796 kB
Inactive(file): 9576412 kB
Inactive(file): 9974336 kB
Inactive(file): 10400772 kB
Inactive(file): 10710464 kB
Inactive(file): 10964180 kB
Inactive(file): 11280900 kB
Inactive(file): 11591336 kB
Inactive(file): 11731164 kB
Inactive(file): 11817340 kB
Inactive(file): 11920016 kB
Inactive(file): 12040800 kB
Inactive(file): 12196232 kB
Inactive(file): 12148272 kB
Inactive(file): 12269224 kB
Inactive(file): 12251768 kB
Inactive(file): 12263596 kB

The number initially ramped up fast, but then leveled off a bit. Just to double check, I dropped the cache again:

Inactive(file): 401152 kB
Inactive(file): 2724788 kB
Inactive(file): 4409916 kB
Inactive(file): 6003208 kB
Inactive(file): 6532220 kB
Inactive(file): 7319768 kB
Inactive(file): 8154560 kB
Inactive(file): 8769084 kB
Inactive(file): 9271760 kB
Inactive(file): 9650020 kB
Inactive(file): 9918932 kB
Inactive(file): 10170456 kB
Inactive(file): 10303404 kB
Inactive(file): 10602256 kB
Inactive(file): 10972760 kB
Inactive(file): 11509680 kB
Inactive(file): 11986980 kB
Inactive(file): 12436528 kB
Inactive(file): 12770672 kB
Inactive(file): 13195352 kB
Inactive(file): 13463276 kB
Inactive(file): 13807816 kB
Inactive(file): 14029160 kB
Inactive(file): 14749976 kB
Inactive(file): 14879704 kB
Inactive(file): 14908984 kB
Inactive(file): 14988196 kB
Inactive(file): 15123316 kB
Inactive(file): 15240824 kB
Inactive(file): 15341328 kB
Inactive(file): 15464332 kB

We got the same behavior, and more importantly, we seem to be reclaiming the memory from Inactive(file). I also checked MemFree and Buffers before/after dropping caches:

(Before)
MemTotal: 66053640 kB
MemFree: 51291028 kB
Buffers: 10685976 kB

(After)
MemTotal: 66053640 kB
MemFree: 63239432 kB
Buffers: 198148 kB

Buffer usage dropped below 200 MB. Given the rate at which we are purging, that never would have happened prior to applying the patch.

It feel 90% confident this patch solved the problem. If we can continue purging at this rate over the couple of days without increased memory usage, then I think I will be 100% confident.

[Peter Jones]

That's great news - thanks Rick!

[Rick Mohr]

Niu,

We were able to get a test file system setup and applied the patch. Initial testing seems to show that the problem is fixed. We have applied the patch to Lustre 2.4.3 and plan to roll it out to our production file system next week. After that, we will run some further tests and let you know if there are any problems.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/13464/
Subject: LU-5726 ldiskfs: missed brelse() in large EA patch
Project: fs/lustre-release
Branch: b2_5
Current Patch Set:
Commit: 8f7181ee5553aa22ecfe51202f3db1a4162361e7

[Rick Mohr]

Niu,

Our testbed is currently down, but we are trying to get it back up and running again. Once that is done, we will work on applying your patch and testing it. (Although this might not happen for another week.)

[Niu Yawei (Inactive)]

Rick, could you verify that if the patch can fix your problem? It works for me, after applied the patch, I didn't see the "growing buffers" problem anymore.

[Rick Mohr]

In response to Andreas' question:

dumpe2fs 1.42.12.wc1 (15-Sep-2014)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent mmp flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota
Journal features: journal_incompat_revoke

Our file system has 90 OSTs.