[LU-10938] Metadata writeback cache support

• Type: New Feature
• Resolution: Unresolved
• Priority: Major

Epic/Theme

Performance

• Assignee: Qian Yingjin
• Reporter: Oleg Drokin

Description

Metadata writeback cache is a highly desirable feature that should allow Lustre to overcome link latency for modifying metadata operations esp. noticeable in interactive type of workloads.

Examples include upacking large archives and building software on a single node with no contention from any other nodes.

Another highly visible set of usecases would include "File/directory-per-process" kind of benchmarks like say mdtest.

Issue Links

• is related to: LU-16361 Lustre client was crashed when I used WBC
• is related to: LU-14139 batched statahead processing
• is related to: LU-13015 WBC: lfs wbc and lctl wbc utils
• is related to: LU-11018 async close RPC
• is related to: LU-14138 Move more members in PTLRPC request into pill
• is related to: LU-13008 WBC: memory limits for caching
• is related to: LU-13009 WBC: fsync() support
• is related to: LU-13010 WBC: Reopen the file when WBC EX lock revoking
• is related to: LU-13015 WBC: lfs wbc and lctl wbc utils
• is related to: LU-13007 WBC: Add cache aging to flush dirty cache in the background
• is related to: LU-13016 WBC1.5: Cached data reintegration after the client was evicted
• is related to: LU-13011 WBC2: Integrate PCC with Metadata Writeback Caching
• is related to: LU-13012 WBC: Rule based auto WBC

Comments

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/37104
Subject: LU-10938 wbc: add symlink support
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 96f7534209e0c7c81b891ea0cadcdb9e9e746237

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/36851
Subject: LU-10938 wbc: Basic test scripts for WBC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 97f122f36e995e251defe8f53a511d614e747ad2

[Patrick Farrell (Inactive)]

I would strongly prefer not merging them.  Having them broken out like this is really nice.

[Qian Yingjin]

I will try to rebase all WBC patches against master branch.
Should I merge the series of WBC patches into a big one patch?

[Qian Yingjin]

After reading the code, I found that current WBC still need to solve the following problem of reopen the file:

1. When WBC is valid, open the file;
2. the EX layout was revoked, the file is flushed
3. at this time, we still need to reopen the file on MDS.

[Nathan Rutman]

Status of the prototype, per ISC

• No cache/quota/space limits, no background flushing, no batching, ...
• Early tests show 10-20x single-client speedup tests (untar, make, …)

[Andreas Dilger]

The timestamps belong to the target that has the most recent ctime, but in any case since this optimization would be useful mostly for DoM files, the MDS would own all of the attributes including the size, as long as the later components were not initialized. Even if the later components were initialized, setting the timestamps, UID, GID, permission, etc. could be cached on the client as long as it has the DoM write lock on the MDT inode, since no other clients could modify/access these attributes.

[Patrick Farrell (Inactive)]

I should perhaps know the answer to this question, but which Lustre attributes are exclusively on the MDS?  Permissions are, xattrs are, and size is not (even in DOM, it's pulled from the data object - it's just that object is on the MDS) but what about the various time properties?

The issue that will come up, I think, is which bits the client has a write lock on.

[Andreas Dilger]

One thought I had was whether it is possible to cache/merge setattr-like requests like utimensat(), fchmod(), fchown(), fchgrp(), etc. on the client when creating/writing DoM regular files on the client? The client should already have a write lock on the file for DoM write, and the MDS shouldn't care whether the time, mode, permission, and maybe UID/GID changes before the file is written to the MDS because these attributes should all be sent with the write request.

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/32241/
Subject: LU-10938 ptlrpc: Add WBC connect flag
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: f024aabf8bbf797e101eec5ff054f7a6e3d253ed