Metadata writeback cache support (LU-10938)

[LU-13008] WBC: memory limits for caching Created: 26/Nov/19  Updated: 10/Jan/22

Status: Open
Project: Lustre
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Technical task Priority: Blocker
Reporter: Qian Yingjin Assignee: Qian Yingjin
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Related
is related to LU-10938 Metadata writeback cache support Open
Rank (Obsolete): 9223372036854775807

 Description   

On a client, WBC borrows the design and implementation from Linux/tmpfs a lot. It uses a virtual in-memory subtree to represent a directory entirely caching on a client.
All inodes and directory entries (dentry for short) are stored in memory and managed by Linux VFS layer, which is a common VFS layer data structure with its own private data.
When create a file under a directory protected by the EX WBC lock, the client only creates the corresponding in-memory inode and dentry, then pins the dentry in the dentry cache by adding the reference to the dentry object.
When unlink the file, put the corresponding reference. Thus, the in-memory dentry and inode will be released when the last reference to the dentry object is deleted.
Also similar to Linux/tmpfs, file data is directly written into and pinned the page caches

To prevent from exhausting all virtual memory on a client, MemFS should allow an administrator to specify a maximum upper bound for the caching size in two aspects:

  • Page cache size for caching file data; 
  • The maximum number of inodes.

Discussion:

Should we add memory cache limit?
Or not limit memory cache, let VM mechanism to reclaim memory cache for WBC automatically (inode/dentry/page caches)? If not limit, it will break the max_cached_mb (llite.*.max_cache_mb) memory usage limit in Lustre...



 Comments   
Comment by Gerrit Updater [ 19/Jan/20 ]

Yingjin Qian (qian@ddn.com) uploaded a new patch: https://review.whamcloud.com/37280
Subject: LU-13008 wbc: inode limits for WBC caching
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 63cc4e4af5e002b3b4c2fcf7afc981bbb6695a5d

Comment by Gerrit Updater [ 20/Apr/20 ]

Yingjin Qian (qian@ddn.com) uploaded a new patch: https://review.whamcloud.com/38279
Subject: LU-13008 wbc: cache page limits for WBC caching
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 3ed6c934c2304c320fd8cebb75ab59e3955072d3

Generated at Sat Feb 10 02:57:34 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.