[LU-14361] Add support for statahead pattern with regularized file name

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.17.0
• Labels: mtg_7 statahead
• Assignee: Qian Yingjin
• Reporter: Qian Yingjin

Description

Some applications do stat() calls under a directory within which all the children files have regularized file name:

• mdtest benchmark: mdtest.$i
• du more than 10000 entries: file.$i
• ML/AI with ingested data that have typically a rule of filename in the directory.

These applications call stat() on the files in the Alphabetic sorting order of the file name.

This kind of metadata operations can not be optimized by current statahead mechanism.
The current statahead mechanism works as follows:
1. Open the directory via opendir() call. It will authorize the statahead.
2. readdir() to get the name and inode number for the dentries.
3. do stat() on the dentries one by one.
4. Close the directory which will deauthorize the statahead.

In current statahead mechanism, the stat() calls in the order of populate the dentries via readdir(). For the ldiskfs backend, it is ordering by the hash of the file name, not a kinds fo sorting oder.

However, we can improve the statahead to support statahead pattern with regularized file name to optimize the metadata performance for the above applications

Issue Links

• is related to: LU-18510 sanity-sec test_63: crash in ll_prep_md_op_data()
• is related to: LU-17450 sanity: interop test failures with master+2.15
• is related to: LU-18536 statahead: decrease the inuse entry count when entry state waiting timeout
• is related to: LU-14139 batched statahead processing
• is related to: LU-17399 sanity: test_123i Error: 'File name pattern statahead not trigger'
• is related to: LU-14139 batched statahead processing
• is related to: LU-14380 Make statahead better support Breadth First Search (BFS) or Depth First Search (DFS)
• is related to: LU-18523 tests sanity/test_65j: wait for ll_sa stat-ahead thread to quit during cleanup

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51587/
Subject: LU-14361 statahead: return ENOENT for batched statahead
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1b7ea3444bf877b4f000c413de1f5473ddf56689

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/54275/
Subject: LU-14361 statahead: add connect flag check for batch RPC
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: fd4c531bbb2af5357d55301801c47ae6b979e5dd

[Gerrit Updater]

"Qian Yingjin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/54275
Subject: LU-14361 statahead: add connect flag check for batch RPC
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 02dadd578a35dff47f107b3c9229ffe6b6b9d336

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53634/
Subject: LU-14361 statahead: increase the initial statahead count
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 76ff18798974b15c4a171dbb76a1077623a66d8a

[Gerrit Updater]

"Qian Yingjin <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53634
Subject: LU-14361 statahead: increase the initial statahead count
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: dd6b7c0f476085c0cbe0c0a4c05710e1205b5d25

[Qian Yingjin]

The reason for these failures may be that @sbi->ll_sa_min is too small, the default value is 

#define LL_SA_REQ_MIN_DEF 8,

it results in the stat-ahead is not so aggressive, too low hit rate and quit wrongly...

Increasing this value in sanity/123i should fix the failure. 

 

I will make a patch later.

[Andreas Dilger]

It looks like the newly-added sanity test_123i from patch https://review.whamcloud.com/41308 "LU-14361 statahead: regularized fname statahead pattern" is failing fairly often since the patch was landed (33/561 = 1/17 runs in the past week). Since the test is in sanity, it is run many times for each patch and much more likely to cause problems:

https://testing.whamcloud.com/search?horizon=2332800&status%5B%5D=FAIL&test_set_script_id=f9516376-32bc-11e0-aaee-52540025f9ae&sub_test_script_id=7d5be5fb-0bc8-4e25-a53f-3a833bca78bd&source=sub_tests#redirect

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/41308/
Subject: LU-14361 statahead: regularized fname statahead pattern
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: e10bf68d7c3eb71a7e84986089c020aa4c9f98f7

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/53173/
Subject: LU-14361 tests: add aheadmany to .gitignore
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 46d69b57d4a2d9eeb4ac7adc472a7667394de919

[Gerrit Updater]

"Timothy Day <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/53173
Subject: LU-14361 tests: add aheadmany to .gitignore
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: 5027f6bc7fceee920ac5a54480db572dbfbcd5bd

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/51572/
Subject: LU-14361 statahead: add tunable for fname pattern statahead
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 6fbffd9c09ad4526ce3a03b8d85f0ce273cac037