"A novity remote shread PCC for" what is "novity" supposed to be? (shread is obviously shared 
)
I think I don't understand the point here - Why is this better than (for example) using Lustre directly to access the back end fs?
The reason is the limited capacity of current node-local PCC. And I think the remote shared PCC is compare favourably with other remote shared burst buffer such as IME and DataWarp, as it has a unified global name space and can transparently access data, etc, al.
"
Thus, it becomes part of the global Lustre namespace. Moreover, the file data can be migrated from the LPCC cache to the Lustre OSTs via file restores, and it is transparent to the application. Furthermore,we can customize various cache strategies and provide cache isolation according to files’ attributes.
"
So one of your examples was using a second Lustre file system, right? So a shared, global file system - Why not access that file system directly?
Is the main idea that you could use this to access files stored on that second file system from the namespace of the first Lustre file system? Hmmmmm!
Before any development effort is spent on this, there are several other things that are more useful to work on, such as productizing the WBC feature, CCI, HSM integration into composite layouts, etc.
I have same question of Patrick asked. "remote PCC" should be Lustre on flash devices right? That means client is able to mount it directy and it should be faster than another PCC layer? Also, if there are mixed SSD and HDD OSTs in same Lustre namespace, FLR is more better way of writing data into SSD layer first, then migrate data to HDD OSTs with FLR mirror, no?
Compared with other remote shared burst buffer (IME and DataWarp, see https://www.nersc.gov/users/computational-systems/cori/burst-buffer/burst-buffer/), I think our remote shared PCC has its advantages:
1) Unified global namespace
2) Any high-speed networked filesystem can be used as PCC backend
3) The file data can be migrated from the PCC cache to the Lustre OSTs via file restores, and it is transparent to the application. While traditional remote shared burst buffer needs to stage data in /out burst buffer cache.
4) we can customize various cache strategies and provide cache isolation according to files’ attributes.
From my understanding, I don't think it needs any development effort (at least it works for RW-PCC without open attach feature/ and RO-PCC with/without open attach eanbled), the current PCC can already be used as remote shared burst buffer.
My preference for long-term development in this area is to have mirrors/copies/archives of data integrated with composite layouts. The way that DAOS is using a foreign layout, and/or moving the HSM xattr into a component to link to copies outside of Lustre seems like the right approach. This allows multiple archive copies per file, and would unify the tools needed to manage PCC, HSM, and FLR.
That would also allow, for example, two Lustre filesystems to link to each other's files (the foreign layout xattr would contain the FID of the remote filesystem copy). In the normal case, each filesystem would have an "archive" copy in the remote filesystem, and if it wants to make a change locally it would mark the local archive copy as being dirty, set the remote primary copy as stale, modify the file locally (similar to PCC), then "archive" the file back to the remote filesystem. This would be symmetrical between both filesystems and allow e.g. remote replication where either copy could be updated (though not both copies at the same time).
>if there are mixed SSD and HDD OSTs in same Lustre namespace, FLR is more better way of writing data into SSD layer first, then migrate data to HDD OSTs with FLR mirror, no?
Compared with FLR, PCC can:
1） Transparently restore data from PCC into Lustre OSTs when it hit -ENOSPC or -EDQUOT error; while FLR on SSD can not tolerate this kind of failures, I think.
2)customize various cache strategies and provide cache isolation according to files’ attributes.
i.e. PCC can provide cache isolation mechanisms for administrators to manage how much PCC storage capacity each user/group/project can use.
FLR can not customize how much SSD space each user/group/project can use.
Pool-based quota maybe, but not in user/group/project dimension, I think.
Moreover, PCC can implement a job-base quota via poject quota on the PCC backend fs, I think.
We just need to add a mapping between the job identifier (i.e. job name) and a dedicated project ID (i.e. 100) of PCC backend fs.
1) Before the job starts, set this mapping and the project quota enforcement on the PCC backend fs;
2) when the job runs, at the time of attaching the file inot PCC, set the project ID (100) to the PCC copy to achive job-base quota.
3) when the job finishes, unmapping the relation between the job identifier and a project ID (i.e. 100) and remove the project quota enforcement associated with this project ID on the PCC backend fs.
The very significant problem with PCC today is that users will LOSE THEIR DATA if the client is offline. Using FLR to keep an extra copy of the data in the PCC cache is much more usable. I don't think FLR and PCC are incompatible with each other, if we move the PCC/HSM xattr into an FLR component. The main difference is that FLR can mark one mirror STALE, but keep the data in the OSTs, while HSM has to release the data permanently. Also, HSM today can only have one archive copy of the data, but if the HSM xattr is moved into a component we could have many copies.
I don't think that any of the PCC quota options are incompatible with Lustre providing quotas itself. Having to manage quota on each PCC node separately/locally would also be complex for the administrator, and not what they want for a distributed filesystem.
I think the first and most important thing to do in this area is to consolidate the HSM xattr with PFL/FLR/composite layouts to give us the flexibility to combine these features in interesting ways. Secondly, PCC is still limited to creating one file on the MDS for each local file, so productizing the WBC feature would allow a client to create files at a high speed locally, without an MDS RPC for each file, which is useful for many things.
Before we add many more complex features to PCC, we also need to get some feedback from users on how it is being used, fix bugs, etc. to know that time spent there is worthwhile compared to other features.