[LU-10499] Readonly Persistent Client Cache support

• Status: Closed
• Project: Lustre
• Type: Technical task
• Priority: Minor
• Reporter: Li Xi
• Assignee: Qian Yingjin
• Resolution: Fixed

Description

Readwrite Persistent Client Cache has been implemented by using HSM features (LU-10092). When a file is cached on Readwrite PCC (RW-PCC), only this single client can read/write to the file. RW-PCC can accelerate I/O intensive applications which read/write data into a file in a single client. However, in some other use cases, part of the files will never be updated, but need to be readed at the same time from multiple clients. Readonly Persistent Client Cache tries to implement a readonly cache level on Lustre clients using SSDs/NVMe.

Readonly Persistent Client Cache (RO-PCC) shares the same framework with Readwrite Persistent Client Cache, except that no HSM mechanism is used in readonly mode of PCC. Instead, readonly mode of PCC uses grouplock to protect the data from being written from other clients.

Whenever a file is fetched into RO-PCC, like the archive number of RW-PCC, a grouplock ID needs to be specified. The grouplock of this ID will be held in the client as long as the file is being cached in RO-PCC. Multiple clients can fetch the same file into its local cache using the same grouplock ID. Because of the semantics of grouplock, no other client can read/write from/to the file if the grouplock with the same ID is not held. Also because of the semantics of grouplock, no client can fetch the file to RO-PCC with a different grouplock ID.

Issue Links

• is blocking: LU-10092 PCC: Lustre Persistent Client Cache
• is related to: LU-13602 Skip unknown FLR component types
• is related to: LU-13842 Can't add pcc cache with roid only pa...
• is related to: LU-13843 removing non existing path with lctl ...
• is related to: LU-13841 error messages need additional explan...
• is related to: LU-10602 Add file heat support for Persistent ...
• is related to: LU-10606 HSM info as part of LOV layout xattr
• is related to: LU-13924 Document PCC "lfs pcc add" options
• is related to: LU-16700 reserve flags and define data structu...

Comments

[Li Xi (Inactive)]

Patch will be pushed as soon as cleanup finishes.

[Gerrit Updater]

Li Xi ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30847
Subject: LU-10499 llite: add readonly support for PCC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a5c1ccfdfa3decb879958cff3de32372a763135b

[Andreas Dilger]

The one problem with the grouplock is that this will be dropped when the filesystem is restarted. Secondly, the grouplock prevents other clients from reading the file, even if there is still a copy on the original OST.

What I think would be a better mechanism for this would be to leverage the FLR functionality and state machine directly for the PCC mirrors.

[Li Xi (Inactive)]

> The one problem with the grouplock is that this will be dropped when the filesystem is restarted. Secondly, the grouplock prevents other clients from reading the file, even if there is still a copy on the original OST.

Agreed. However, I don't think dropping the lock when the file system is restarted is a big problem. Remounting the client doesn't happen a lot.

In order to solve the problem that other clients can't reading the PCC cached files, we can either:

1) Add readonly support for the grouplock. When some on is holding the readonly grouplock of the file, no write lock is allowed, and only readlock can be granted.

2) Add none-copy support for readonly PCC. When a client is accessing the file which is cached by other clients on readonly PCC, the client will require the PCC grouplock automatically, but will not copy the data to local storage. Instead, the client will read from the OSTs directly.

Either way should work well. But I'd prefer solution 2), unless solution 1) can help other use cases since solution 1) will be much harder.

> What I think would be a better mechanism for this would be to leverage the FLR functionality and state machine directly for the PCC mirrors.

I don't have any idea about how to use FLR for PCC. Can you explain more details?

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/31868
Subject: LU-10499 llite: Add new DLM mode PCCRO for readonly PCC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: a6e8e5b5108168cfb907f23122ce2ef0c6a37822

[Qian Yingjin (Inactive)]

 Currently Readonly mode of PCC uses grouplock to protect the data from being written or even normal read from other clients, even if there is still a copy on the original OSTs.
The patch above introduces a new kind of LDLM lock mode PCCRO, which is compatible with PR lock mode (but not compatible with PW and GROUP lock mode), so when the file is PCC-fetched and attached on a client, the client will obtain PCCRO lock on the file, the PCCRO lock is kept until the administrator detachs the PCC-fetched file. It allows the read access for the file data on original OSTs from other clients, but no write lock (PW) is allowed.

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32021
Subject: LU-10499 llite: fix problem of fetch
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 83af1ef273fe694df1fbbf7d4cc3cbb33f803194

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32852
Subject: LU-10499 pcc: add readonly mode for PCC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 952936bcf729ff8e0a8c013bebe4ebc218bda280

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/32998
Subject: LU-10499 pcc: add readonly mode for PCC
Project: fs/lustre-release
Branch: pcc
Current Patch Set: 1
Commit: d8ccc3ef8bbf7c3f180681dd758833875733956e

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/38305
Subject: LU-10499 pcc: add readonly mode for PCC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 22e483f35b11758575fe1d752cf0d40d4e09315f

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/39483
Subject: LU-10499 hsm: make HSM layout be a kind of foreign layout
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 10e6228d530df034513d6a41f24f28243e13bae2

[Gerrit Updater]

Yingjin Qian ([e-mail]) uploaded a new patch: https://review.whamcloud.com/40791
Subject: LU-10499 pcc: introducing OBD_CONNECT2_PCCRO flag
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: d2638ccd5c1a327840c68d875eb30a0a3f4f895d

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/40791/
Subject: LU-10499 pcc: introducing OBD_CONNECT2_PCCRO flag
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6007dc9382df7260841a4748158307ade25f22ef

[Colin Faber]

Per lixi_wc  – all the known issues of Readonly PCC have been solved.

[Gerrit Updater]

"Li Xi <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/47887
Subject: LU-10499 pcc: add readonly mode for PCC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: cbe025c527067f894705df1fe55dccc3cade511a

[Gerrit Updater]

"Qian Yingjin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/51375
Subject: LU-10499 pcc: use foreign layout for PCCRO on server side
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 22dc3585644251aeae2042b6a29f7846a982cbca

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51375/
Subject: LU-10499 pcc: use foreign layout for PCCRO on server side
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: aebb405e32e4d40bc2d70a95356b3b85be280668