[LU-19598] TCU: Clean up files from the Trash Can

• Type: Technical task
• Resolution: Unresolved
• Priority: Medium
• Severity: 3
• Assignee: Emoly Liu
• Reporter: Qian Yingjin

Description

A mechanism is needed to automatically clean up files from the trash can when the filesystem becomes full or the files in Trash Can are too old to expire the retain period.

• For emergency case, potentially via MDS kernel thread to skip moving files into Trash Can (destroy the file directly), or delete the oldest pFID tree;
• Userland policy engine can implement more sophisticated policy(age/space by user/projid, or file name, etc). It monitors the space/inode usage on the MDTs and OSTs periodically, once found the space is nearly full:
  • Scan MDT or OST devices for files in Trash Can via policy engine
  • Check TRASH and NOTRASH attributes on file (bot MDT and OST)
  • Check the original UID/GID/PROJID and deletion timestamp in "trusted.unrm" XATTR
  • Choose the candidate files or directories to be purged from Trash Can to free up space;

To improve the performance, we can use multiple threads to do file purge or improve the mpiFileUtils to scan and purge the files in Trash Can in parallel.

Issue Links

• is related to: LU-19969 ltrash_purge should wait for next scan if usage is below watermark in daemon mode
• is related to: LU-19950 TCU: ENOSPC-triggered ltrash_purge upcall
• is related to: LU-20060 Nodemap policy support in ltrash_purge
• is related to: LU-20148 Purge directories in parallel in ltrash_purge
• is related to: LU-20284 Add --force option to ltrash_purge
• is related to: LU-20092 ltrash_purge: always dump stats data when exiting if a dump file is specified
• is related to: LU-18456 TCU: Trash Can/Undelete for Lustre