Metadata writeback cache support (LU-10938)

[LU-13045] WBC2: batch metadata update Created: 04/Dec/19  Updated: 13/Feb/23

Status: Open
Project: Lustre
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Technical task Priority: Major
Reporter: Qian Yingjin Assignee: Qian Yingjin
Resolution: Unresolved Votes: 0
Labels: None

Issue Links:
Related
is related to LU-14139 batched statahead processing Resolved
Rank (Obsolete): 9223372036854775807

 Description   

During flush under WBC, a client could send a batch of metadata updates to MDT to improve the metadata performance a lot.

 

For batch metadata updates, the following points need to consider:

  • If batch metadata updates need to return with locks (in case of WBC EX lock revocation), it needs to grant a batch of locks to the client and the client needs to cache the returned batch of locks on its DLM lock namespace. 
  • Transaction:
  • Recovery: VBR
  • Dependency:
    • The client must ensure the dependencies in batching metadata updates must be ordered.
    • Each metadata update action in batching metadata update request should be executed one by one in a single thread on MDT;
    • If want to improve the performance of batching metadata update request by executing metadata updates in parallel in multiple threads, the MDT must carefully handle the dependencies.
    • When DNE involves, the client also needs to consider the dependencies among MDTs when do a batching metadata updates which is same as boosting batching metadata updates using multiple threads..

 

 

 Comments   
Comment by Gerrit Updater [ 13/Nov/20 ]

Yingjin Qian (qian@ddn.com) uploaded a new patch: https://review.whamcloud.com/40644
Subject: LU-13045 wbc: batch metadata processing framework
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: c176ab73dcf8e5287e2e0ede5200ad85709fb61b

Comment by Gerrit Updater [ 27/May/21 ]

Yingjin Qian (qian@ddn.com) uploaded a new patch: https://review.whamcloud.com/43839
Subject: LU-13045 wbc: reconstruct the code
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 45ec308dffa83f86c6410f5ddf37671bf1173706

Comment by Gerrit Updater [ 20/Jul/21 ]

Yingjin Qian (qian@ddn.com) uploaded a new patch: https://review.whamcloud.com/44352
Subject: LU-13045 wbc: use IO context for various I/O flush engine
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 0228415006e30df4c1335a6dcfe5ffeb23c7e977

Generated at Sat Feb 10 02:57:53 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.