[LU-14150] Extreme create scaling: batching FID alloc, batching creation, batching MD returning to the client

• Status: Open
• Project: Lustre
• Type: New Feature
• Priority: Minor
• Reporter: Qian Yingjin
• Assignee: Qian Yingjin
• Resolution: Unresolved

Description

As we known, some application creates a number of files under a directory with a dedicated file name format, i.e. mdtest creates files with the format:

char curr_item[MAX_PATHLEN];

sprintf(curr_item, "%s/file.%s"LLU"", path, mk_name, itemNum);

i.e. file.mdtest.$i

Lustre have already implemented a basic framework for batching metadata processing.
It also can implement create_many with batching.
For this goal, it needs to:
1) Allocate a FID range within a sequence (belong to a same MDT) for batching creation.
2) filename format parsing, i.e. file.mdtest.$i or file.$jobid.$i
3) MDT does batch the file creations under a directory: unpack the sub creation request in the batch; execute the create request; pack the reply message with MD and other information returning to the client.

It can provide some llapi or lfs command (lfs create_many -F mdtest_file.$i?) to do batch creations, which could improve the performance for metadata operations.

We can also improve the mdtest to support batch creation interface.