[LU-14361] Add support for statahead pattern with regularized file name

• Type: Improvement
• Resolution: Unresolved
• Priority: Minor
• Fix Version/s: Lustre 2.17.0
• Labels: statahead
• Assignee: Qian Yingjin
• Reporter: Qian Yingjin

Description

Some applications do stat() calls under a directory within which all the children files have regularized file name:

• mdtest benchmark: mdtest.$i
• du more than 10000 entries: file.$i
• ML/AI with ingested data that have typically a rule of filename in the directory.

These applications call stat() on the files in the Alphabetic sorting order of the file name.

This kind of metadata operations can not be optimized by current statahead mechanism.
The current statahead mechanism works as follows:
1. Open the directory via opendir() call. It will authorize the statahead.
2. readdir() to get the name and inode number for the dentries.
3. do stat() on the dentries one by one.
4. Close the directory which will deauthorize the statahead.

In current statahead mechanism, the stat() calls in the order of populate the dentries via readdir(). For the ldiskfs backend, it is ordering by the hash of the file name, not a kinds fo sorting oder.

However, we can improve the statahead to support statahead pattern with regularized file name to optimize the metadata performance for the above applications

Issue Links

• is related to: LU-18510 sanity-sec test_63: crash in ll_prep_md_op_data()
• is related to: LU-18536 statahead: decrease the inuse entry count when entry state waiting timeout
• is related to: LU-17450 sanity: interop test failures with master+2.15
• is related to: LU-14139 batched statahead processing
• is related to: LU-17399 sanity: test_123i Error: 'File name pattern statahead not trigger'
• is related to: LU-14139 batched statahead processing
• is related to: LU-14380 Make statahead better support Breadth First Search (BFS) or Depth First Search (DFS)
• is related to: LU-18523 tests sanity/test_65j: wait for ll_sa stat-ahead thread to quit during cleanup