[LU-18456] Recycle Bin for Lustre

• Type: New Feature
• Resolution: Unresolved
• Priority: Minor
• Severity: 3
• Assignee: Qian Yingjin
• Reporter: Qian Yingjin

Description

Introduction

If files are accidentally deleted from a file system, an application may be interrupted and the user data may be permanently lost. The recycle bin is a recommended feature in file systems that acts as a virtual trash can, allowing users to store deleted files temporarily before permanently deleting them. It provides a way to restore or retrieve deleted files if needed.

Once the recycle bin feature is enabled, when a user deletes a file from a file system, it is not actually deleted but moved to the recycle bin, deleted files and directories are temporarily stored in the recycle bin. The recycel bin may be manually emptied or once it is full, it will remove the oldest files first. Additionally, items in the recycle bin may be restored or retrieved if they are still there.

Recycle Bin Functionalities

The recycle bin should including the following functionalities:

• List "undeleted" files in the recycle bin;

• After a file is deleted and moved into recycle bin, the quota for this file should be accounted and updated (reduced) accordingly;

• A file in the recycle bin is not visiable in the namespace of the file system;

• Restore a file in the recycle bin. This will restore a file to its original path. The corresponding quota account should be updated also;

• Delete a file in the recycle bin. This will finally remove the file from the file system and free the used space. The file is now unrecoverable;

• Empty the recycle bin. This will remove all files in the recycle bin;

• A user can restore files from recycle bin within the specified retention period. By this way, a file can be kept "undeleted" under a pre-defined configureable grace period.

• Enable/disable recycle bin feature on a entire file system;

• A administrator can enable/disable recycle bin feature on a specified directory;

Deleted files can no longer be restored from the recycle bin when:

• A file (or directory) is deleted again from the recycle bin. In other words it have been deleted twice. The first deletion only moves the file to the recycle bin. The second deletion actually removes the file from the file system.

• The recycle bin is emptied of all of its contents.

Design and Implementation for Recycle Bin in Lustre

The design for the recycle bin feature in Lustre is simple.

On the server side, It just implements the basic functionalities such as moving the "undeleted" files into the cycle bin and the interface how to traverse them. On the client side, it implements the basic utility tools to interact with the recycle bin (lctl recycle set|clear|list|delete|restore xxx), including:

• Set or clear the recycle flag on a given file or directory;
• list "undeleted" files on a given MDT;
• Permantently delete a file within the recycle bin on a given MDT;
• Empty the recycle bin on a give MDT;
• Restore a file within the recycle bin on a give MDT;

Our mechanism only moves the regular files into the recycle bin upon its last unlink, but ignoring the directories.

It borrows lots of ideas from orphan and volatile files in Lustre (which stores in "ROOT/PENDING"" directory on each MDT). During the format and setup, each MDT creates a "ROOT/RECYCLE" directory as a recycle bin to store "undeleted" files.

The POSIX API is used to traverse the files under the recycle bin on a given MDT. First, a client can get the FID of recycle bin directory "ROOT/RECYCLE" on the MDT. Then the client can get the file handle via dir_fd=llapi_open_by_fid(); after that, the "undeleted" files within the recycle bin can be traversed via readdir(); it can open by openat(dir_fd, dent) and obtain the "undeleted" XATTR, which contains the necessary information to resotre, via fgetxattr(fd, "trusted.recyclebin"); The client can even read the data or swap layout of the "undeleted" file on the recycle bin for restore: opendir()/readddir()/openat()/fgetxattr("trusted.recyclebin")/close()/closedir();

The workflow for the recycle bin is as follows:

• An administrator can enable/disable recycle bin feature on a specified MDT via: mdd.*.recycle_bin_enabled;
• An adminstrator can enable/disable recycle bin feature on a specified directory or a file via the Lustre specified file flag: LUSTRE_RECYCLE_FL (similar to LUSTRE_ENCRYPT_FL); All sub files under a directory flagged with LUSTRE_RECYCLE_FL can inherit this flag;

 # lctl recycle set_flag $file|$dir
 # lctl recycle clear_flag $file|$dir

• Move a deleting file into the recycle bin. When delete a regular file marked with LUSTRE_RECYCLE_FL upon its last unlink, first move the file into the recycle bin directory "ROOT/RECYCLE" with FID as its name. And then set a "trusted.recyclebin" XATTR on the "undeleted" file on the recycle bin. The XATTR contains the following information:

struct recycle_bin_xattr {
 __u32 uid; // uid of the deleting file, used for quota accounting
 __u32 gid; // gid of the deleting file, used for quota accounting
 __u32 projid; //projid of the deleting file, used for quota accounting
 __u64 timestamp; // Timestamp that the file moved into the recycle bin
 char fullpath[MAX_PATH]; // Full path of the deleting file
};

Where uid/gid is the original uid/gid of the deleting file, mainly used for quota accounting for the restore operation; @timestamp is the time that the file was moved into the recycle bin. It is used to determine whether the file is expired for the specified retention period and thus should be removed from the recycle bin finially; @fullpath is the full path of the deleting file and is used for restoring the file metadata. During deleting the file, we can get the full path information via the way similar to fid2path().

• List "undeleted" files within a recycle bin on a given MDT:

   # lctl recycle list <--mdt|-m mdt> MNTPT
  MDT index: 1
  uid gid size delete time FID Fullpath
  0 0 4096 Nov 14 08:11 [0x200034021:0x1:0x0]->/mnt/lustre/f1
  0 0 32104 Nov 14 08:07 [0x200034021:0x2:0x0]->/mnt/lustre/dir/f2
  ...
  

The preudo code:

rbin_fid = llapi_recycle_fid_get(MNTPT, mdt);
dir_fd = llapi_open_by_fid(MNTPT, rbin_fid);
while ((ent = readdir(dir_fd)) != NULL) {
    fd = openat(dir_fd, ent->d_name);
    fgetxattr(fd, "trusted.recycle", xattr_buf);
    print_one(ent->d_name, xattr_buf);
    close(fd);
}
close(dir_fd);

• Deleting a file in the recycle bin will remove the temporary file under "ROOT/RECYCLE" and free the data space on Lustre OSTs permantently.

   # lctl recycle delete <--mdt|-m mdt> MNTPT FID
  

  The pseudo code:

  rbin_fid = llapi_recycle_fid_get(MNTPT, mdt);
  dir_fd = llapi_open_by_fid(MNTPT, rbin_fid);
  unlinkat(dir_fd, "FID", 0);
  close(dir_fd);
  

• Empty a recycle bin:

 # lctl recycle clear <--mdt|-m mdt> MNTPT FID

The pseudo code:

rbin_fid = llapi_recycle_fid_get(MNTPT, mdt);
dir_fd = llapi_open_by_fid(MNTPT, rbin_fid);
while ((ent = readdir(dir_fd)) != NULL) {
    unlinkat(dir_fd, ent->d_name, 0);
}
close(dir_fd);

• Resotre a file in the recycle bin on a given MDT. It will restore the file and its content according to the saved full path and then delete the stub on the recycle bin.

 # lctl recycle restore <--mdt|-m mdt> MNTPT FID

The pseudo code:

rbin_fid = llapi_recycle_fid_get(MNTPT, mdt);
dir_fd = llapi_open_by_fid(MNTPT, rbin_fid);
fd = openat(dir_fd, FID, O_RDONLY);
fgetxattr(fd, "trusted.recycle", xattr_buf);
mkdir -p dirname(xattr_buf.path)
{ way 1: 
dst_fd = open(xattr_buf.path, O_CREAT);
// copy the file data via read()/write() syscall
copy_data(dst_fd, fd);
close(dst_fd);
unlinkat(dir_fd, "FID", 0);
} 
{ way 2:
mknod(xattr_buf.path);
dst_fid=path2fid(xattr_buf.path)
swap_layouts(dst_fid, FID);
unlinkat(dir_fd, "FID", 0)
}
{ way 3:
parent_fid=path2fid(dirname(xattr_buf.path))
ioctl(IOCTL_RECYCLE_RESTORE, parent_fid, FID);
in the ioctl(), mv the FID into parent_fid on MDT.
}
close(fd);
close(dir_fd);

• LFSCK periodically scans the files under recycle bin directory "ROOT/RECYCLE" and delete the file with grace time expired.

• Provide the functionality to scan "undeleted" files on all MDTs with the grace time expired manually and delete all of them.

  # lctl recycle check [--expire_time|-E time] MNTPT
  

• Provide the functionality to restore/delete all files within a given directory.
  This can be achieved by using the command combination of "lctl recycle list" and "lctl recycle restore" or "lctl recycle delete" to filter the files with the full path attribute within a given directory.

• Provide ".recycle/mdt[N]" (where N is the MDT index) filesystem namespace. By this way, users can access the "undeleted" files with readonly mode under the recycle bin directory on a given MDT[N] via POSIX file system API. However, we can not access these files from fileset sub directory mount. We can perform the following commands from a Lustre namespace (mount point of "/mnt/lustre") on a client:

# ls /mnt/lustre/.recycle/mdt2
0x200034021:0x1:0x0
0x200034021:0x2:0x0
...

# cat /mnt/lustre/.recycle/mdt2/0x200034021:0x1:0x0
# lctl recycle info /mnt/lustre/.recycle/mdt2/0x200034021:0x1:0x0
0 0 4096 Nov 14 08:11 [0x200034021:0x1:0x0]->/mnt/lustre/f1
# lctl recycle list /mnt/lustre/.recycle/mdt2
MDT index: 1
uid gid size delete time FID Fullpath
0 0 4096 Nov 14 08:11 [0x200034021:0x1:0x0]->/mnt/lustre/f1
0 0 32104 Nov 14 08:07 [0x200034021:0x2:0x0]->/mnt/lustre/dir/f2
...

Issue Links

• duplicates: LU-18253 implement Trash Can/Undelete
• is related to: LU-18457 Flashback for Lustre