[LU-10938] Metadata writeback cache support

• Type: New Feature
• Resolution: Unresolved
• Priority: Major

Epic/Theme

Performance

• Assignee: Qian Yingjin
• Reporter: Oleg Drokin

Description

Metadata writeback cache is a highly desirable feature that should allow Lustre to overcome link latency for modifying metadata operations esp. noticeable in interactive type of workloads.

Examples include upacking large archives and building software on a single node with no contention from any other nodes.

Another highly visible set of usecases would include "File/directory-per-process" kind of benchmarks like say mdtest.

Issue Links

• is related to: LU-16361 Lustre client was crashed when I used WBC
• is related to: LU-14139 batched statahead processing
• is related to: LU-13015 WBC: lfs wbc and lctl wbc utils
• is related to: LU-11018 async close RPC
• is related to: LU-14138 Move more members in PTLRPC request into pill
• is related to: LU-13008 WBC: memory limits for caching
• is related to: LU-13009 WBC: fsync() support
• is related to: LU-13010 WBC: Reopen the file when WBC EX lock revoking
• is related to: LU-13015 WBC: lfs wbc and lctl wbc utils
• is related to: LU-13007 WBC: Add cache aging to flush dirty cache in the background
• is related to: LU-13016 WBC1.5: Cached data reintegration after the client was evicted
• is related to: LU-13011 WBC2: Integrate PCC with Metadata Writeback Caching
• is related to: LU-13012 WBC: Rule based auto WBC

Comments

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37104
Subject: LU-10938 wbc: add symlink support
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 96f7534209e0c7c81b891ea0cadcdb9e9e746237

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36851
Subject: LU-10938 wbc: Basic test scripts for WBC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 97f122f36e995e251defe8f53a511d614e747ad2

[Patrick Farrell (Inactive)]

I would strongly prefer not merging them.  Having them broken out like this is really nice.

[Qian Yingjin]

I will try to rebase all WBC patches against master branch.
Should I merge the series of WBC patches into a big one patch?

[Qian Yingjin]

After reading the code, I found that current WBC still need to solve the following problem of reopen the file:

1. When WBC is valid, open the file;
2. the EX layout was revoked, the file is flushed
3. at this time, we still need to reopen the file on MDS.

[Nathan Rutman]

Status of the prototype, per ISC

• No cache/quota/space limits, no background flushing, no batching, ...
• Early tests show 10-20x single-client speedup tests (untar, make, …)