[LU-10530] Scanning tool for HSM storage of POSIX copytool

Blocker — Thu, 18 Mar 2021 02:25:17 +0000

The current HSM POSIX copytool works well as a demo of the copytool implementation. However, the copytool will create a lot of directoy entries under the HSM storage, which will never be cleaned up by the copytool. Also, if Remove Archive on Last Unlink policy (~~LU-4640~~) is not enabled and policy engine like Robinhood is not used, then a lot of orphan files will be left there in HSM storage. Thus, we want to develop a tool which will scan the directory tree of the HSM storage and cleanup the directory three.

Also, since PCC (~~LU-10092~~, ~~LU-10499~~) also uses the HSM directory structure, this tool could aslo be used for PCC to:

1) scan the whole PCC storage to detach all the cached files when it is not suitable any more to cache the files on PCC, e.g. when a job stops and another job is starting to use PCC.

2) apply some kind of policy (e.g. based on UID/GID/JOBID/ProjID, etc) for PCC cache management.

These use cases needs full scanning of the HSM storage directory, and that is what we want to implement.

Whamcloud Community JIRA

[LU-10530] Scanning tool for HSM storage of POSIX copytool